Three-dimensional scene reconstruction method based on intelligent LED street lamp multi-mode sensor

Abstract

The invention discloses a three-dimensional scene reconstruction method based on an intelligent LED street lamp multi-mode sensor. The three-dimensional scene reconstruction method comprises the following steps that RGB images are obtained and preprocessed; the information is input to a visual feature coding module, two-dimensional bounding box information is extracted, and an object segmentation module is guided to output a two-dimensional segmentation mask; obtaining point cloud data, projecting the point cloud data to the standardized RGB image, and screening target points in combination with the two-dimensional segmentation mask; complementing the preliminary segmentation result of the point cloud, mapping the result to a standardized RGB image, and extracting a pixel region; carrying out joint coding, implicit representation and neural decoding processing on the object-level RGB image and the point cloud complete segmentation result; and fusing into an original three-dimensional scene, and completing spatial restoration through point cloud registration, attitude optimization and semantic constraint. The invention provides an efficient three-dimensional scene reconstruction method in combination with a multi-mode sensor of an intelligent LED street lamp, and the method has high precision, real-time performance and dynamic target processing capability.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional scene reconstruction, and in particular to a three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light. Background Technology

[0002] Currently, 3D scene reconstruction technology mainly relies on devices such as LiDAR and multi-view camera arrays. These methods can provide high-precision spatial data and are widely used in fields such as autonomous driving, urban planning, and virtual reality. Traditional 3D reconstruction methods, such as those based on structured light, stereo vision, or motion reconstruction algorithms, can obtain relatively accurate models in static environments. However, these methods have significant limitations in dynamic environments, especially when dealing with fast-moving objects in complex scenes such as urban roads. For scenes with occlusion or dynamic objects, these techniques struggle to guarantee high-precision reconstruction results and often cannot process large-scale 3D data in real time.

[0003] Existing technologies suffer from limitations in the accuracy of point cloud and image data fusion, particularly in multimodal data fusion, dynamic object modeling, and extraction of scene geometric details. Current methods cannot effectively address the issues of incomplete point cloud data or geometric distortion caused by changes in viewpoint and sensor occlusion, resulting in 3D reconstruction quality failing to meet practical application requirements in dynamic environments. Furthermore, traditional methods suffer from high computational complexity and poor real-time performance when reconstructing large-scale urban scenes.

[0004] Therefore, how to provide a three-dimensional scene reconstruction method based on a multimodal sensor of intelligent LED streetlights is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light. This invention combines the multimodal sensor of an intelligent LED street light to propose an efficient three-dimensional scene reconstruction method with high precision, real-time performance, and dynamic target processing capabilities.

[0006] A three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light according to an embodiment of the present invention includes the following steps:

[0007] Acquire RGB images, perform preprocessing, and generate standardized RGB images;

[0008] The standardized RGB image is input into the visual feature encoding module to extract the two-dimensional bounding box information of the candidate object, and guide the object segmentation module to generate the two-dimensional segmentation mask of the corresponding target object.

[0009] Point cloud data is acquired, projected onto the pixel coordinate system of a standardized RGB image using a rigid extrinsic matrix and a camera intrinsic matrix, and target points are filtered using a two-dimensional segmentation mask to generate preliminary point cloud segmentation results.

[0010] Based on the parallax information of multiple LiDAR sensors, the preliminary point cloud segmentation results are completed at the object level to generate a complete point cloud segmentation result. The three-dimensional spatial coordinate information of each target point in the complete point cloud segmentation result is mapped to a standardized RGB image, and the corresponding pixel region is extracted to generate an object-level RGB image.

[0011] Using object-level RGB images and complete point cloud segmentation results as input, a generative model based on implicit representation is used for joint encoding, implicit representation and neural decoding to generate object-level 3D Mesh results.

[0012] The object-level mesh results are fused into the original 3D scene. By aligning the position and orientation through point cloud registration, pose optimization, and semantic constraints, the spatial reconstruction of the target object in the 3D scene is completed, generating a complete 3D scene.

[0013] Optionally, the preprocessing includes normalizing pixel values ​​and adjusting the image size to a preset input resolution.

[0014] Optionally, the generation of the two-dimensional segmentation mask specifically includes:

[0015] The standardized RGB image is input into the visual feature encoding module to extract multi-scale image feature representations containing spatial semantic information. The visual feature encoding module adopts a multi-scale hierarchical structure based on visual Transformer. By inputting the standardized RGB image, it generates a multi-scale image feature map containing global semantics and local details.

[0016] The multi-scale feature representation of the image is input into the object detection module, which outputs the category label and corresponding two-dimensional bounding box information for each object. The object detection module is built based on the pre-trained object detection Transformer architecture and uses a multi-head self-attention mechanism to perform semantic parsing of potential objects in the image. The two-dimensional bounding box information consists of a quadruple of center x-coordinate, center y-coordinate, bounding box width, and bounding box height in the image coordinate system.

[0017] The two-dimensional bounding box information is input into the object segmentation module, which guides the object segmentation module to generate a two-dimensional segmentation mask for the corresponding target object within each bounding box region. The object segmentation module is constructed based on the Segment Anything model, and the two-dimensional segmentation mask is a binary image with the same resolution as the standardized RGB image.

[0018] Optionally, the generation of the preliminary point cloud segmentation result specifically includes:

[0019] Point cloud data is acquired, and the spatial coordinate information of all three-dimensional points is extracted. The point cloud data is collected by a LiDAR sensor installed on a smart LED street light. The spatial coordinate information includes three parts: horizontal coordinate, vertical coordinate, and height coordinate.

[0020] Based on the extrinsic calibration results, the spatial coordinate information of each three-dimensional point in the point cloud data is projected onto the two-dimensional pixel coordinate system of the standardized RGB image using the rigid extrinsic matrix between the camera and LiDAR and the camera intrinsic matrix, so as to obtain the corresponding pixel coordinate information.

[0021] The external parameter calibration result refers to the rigid transformation matrix that describes the three-dimensional spatial geometric relationship between the camera and LiDAR, which is calculated based on the common reference coordinate system of the two sensors after the camera and LiDAR on the smart LED street light respectively collect the image and point cloud data of the same checkerboard. The camera's attitude information relative to the checkerboard is calculated and extracted using the Zhang Zhengyou calibration method, and the LiDAR's attitude information relative to the checkerboard is extracted using the point cloud plane fitting method.

[0022] The rigid extrinsic parameter matrix is ​​the transformation matrix of the three-dimensional spatial geometric relationship between the camera and the LiDAR, and it consists of a rotation matrix and a translation vector.

[0023] The camera intrinsic parameter matrix is ​​a set of intrinsic parameters of the camera imaging model, specifically including focal length, principal point coordinates, and pixel scaling factor;

[0024] The pose parameters include a rotation matrix and a translation vector;

[0025] The pixel coordinates of each 3D point are judged. If the pixel value at the corresponding position in the 2D segmentation mask is a foreground value, it is determined to be a target point. If the value is a background value, it is discarded. The foreground value is the pixel value of the pixel region of the target object in the 2D segmentation mask, which is usually set to 1. The background value is the pixel value of the pixel region of the non-target region in the 2D segmentation mask, which is usually set to 0.

[0026] By combining all the 3D points identified as target points, a preliminary point cloud segmentation result constituting the target object is generated.

[0027] Optionally, the generation of the object-level RGB image specifically includes:

[0028] Perform normal vector analysis and curvature estimation on the three-dimensional spatial coordinate information in the preliminary point cloud segmentation results, detect edge regions with abrupt changes in normal vectors or a sudden drop in point density, and generate missing regions.

[0029] Point cloud data collected by multiple LiDAR sensors is extracted to generate multi-view point cloud data. Based on the rigid extrinsic matrix in the extrinsic calibration results, the multi-view point cloud data is uniformly transformed to the common three-dimensional coordinate system of the target object.

[0030] Based on multi-view point cloud data, the depth difference between each view is calculated and used as a constraint. Combined with the N-ICP registration method, the multi-view point cloud data is aligned to the same spatial coordinate system. During the registration process, the matching relationship between point clouds is optimized through multiple rounds of iteration to solve the geometric discontinuity problem caused by occlusion or view difference. For missing regions, interpolation methods are used to fill in the missing regions. The point cloud data of the missing regions is inferred using the geometric features and depth information of the surrounding point clouds. Finally, fused multi-view point cloud data is generated.

[0031] Based on the fusion of multi-view point cloud data, local surface fitting processing is performed on the boundary region to improve the overall geometric continuity and boundary smoothness of the point cloud, and output the complete segmentation result of the point cloud. The boundary region is the location where there is still geometric discontinuity in the fusion of multi-view point cloud data.

[0032] Based on the spatial coordinate information of each target point in the complete point cloud segmentation result, and combined with the rigid extrinsic matrix between the camera and LiDAR and the camera intrinsic matrix, the three-dimensional spatial coordinate information of the target point is projected onto the two-dimensional pixel coordinate system of the standardized RGB image, the corresponding pixel region is extracted, and an object-level RGB image is generated.

[0033] Optionally, the generation of the object-level Mesh result specifically includes:

[0034] The complete point cloud segmentation result is input into the point cloud encoding module of the generative model based on implicit representation. Spatial structural features are extracted using a sparse 3D convolutional network to generate implicit geometric representation vectors.

[0035] The implicit representation-based generative model refers to a neural network model that combines point cloud and image information for 3D mesh reconstruction. Based on the original 3DShape2VecSet method, an image-guided multimodal coding structure is designed. By encoding image features into spatial embedding vectors and fusing them with the implicit representation of point clouds, the joint expression of texture and geometric information is achieved. Specifically, it includes a point cloud coding module, an image coding module, a feature fusion module, an implicit field modeling module, and a texture mapping module.

[0036] The object-level RGB image is input into the image encoding module to extract its multi-scale semantic information and local texture features, and then encoded into an image guided representation vector through a spatial embedding mechanism.

[0037] The image guided representation vector and implicit geometric representation vector are input into the feature fusion module. The two are weighted and calculated through a cross-modal attention mechanism to automatically learn the relationship between image texture information and geometric information. By calculating similarity and difference, key features are extracted and a unified multimodal joint feature representation is generated.

[0038] The multimodal joint feature representation is input into the implicit field modeling module. Based on the continuous spatial coordinate query mechanism, the neural decoding network predicts the spatial occupancy probability of each three-dimensional spatial sampling point. The continuous spatial coordinate query mechanism refers to the process of implicit field modeling in which any continuous coordinate point in three-dimensional space is used as input, and the joint feature representation vector obtained by multimodal fusion is used to guide the neural decoding network to predict the spatial occupancy status of each continuous coordinate point. Based on the coordinate perception neural network, the three-dimensional spatial sampling point is concatenated with the multimodal feature vector and input into the multilayer perceptron. The coordinate position is calculated point by point to determine whether it is on the surface of the object. Thus, a continuously defined implicit field function is constructed in the entire three-dimensional space to realize Mesh extraction.

[0039] An implicit field function is constructed based on the spatial occupancy probability. Triangular patches are extracted using the differentiable Marching Cubes method to generate a structurally continuous triangular mesh model. The differentiable Marching Cubes method refers to dividing the constructed continuous implicit field function into voxels based on the implicit field modeling, and judging its topological structure in each voxel cell according to the spatial occupancy probability of the sampling point, thereby extracting isosurfaces to form a triangular mesh.

[0040] The image-guided representation vector is input into the texture mapping module. Based on the spatial projection relationship between the triangular mesh and the image, the image texture is mapped onto the surface of the triangular mesh to generate an object-level 3D Mesh result with texture information.

[0041] Optionally, the generation of the complete 3D scene specifically includes:

[0042] Extracting the object-level 3D Mesh result and the background point cloud data in the original 3D scene, performing initial coarse registration of the object-level 3D Mesh model based on the spatial geometric distribution relationship, and calculating the initial pose parameters in the original 3D scene. The initial pose parameters include rotation matrix and translation vector. The background point cloud data in the original 3D scene refers to the set of 3D spatial points that were not identified as target points during the initial segmentation and completion of the point cloud.

[0043] Point cloud registration error, physical contact constraint, and semantic consistency constraint are used as loss function terms to construct a pose optimization objective function. The spatial position and pose of the object-level 3D Mesh model are optimized by minimizing the pose optimization objective function. The point cloud registration error term refers to the geometric deviation between the object-level 3D Mesh model and the background point cloud data in 3D space. The physical contact constraint term refers to the distance error between each contact point pair between the bottom point set of the Mesh model and the ground area in the background point cloud data, which quantifies the contact relationship and is ultimately used as part of the optimization objective function. The semantic consistency constraint term is formed by calculating the rotation error and position error of the target object and then weighting and summing them to form the loss function term.

[0044] The non-rigid pose optimization method is used to iteratively adjust the pose parameters of the object-level 3D Mesh model. By performing multiple rounds of optimization and iteration on the pose optimization objective function, the rotation matrix and translation vector are updated to generate the pose-optimized object-level 3D Mesh model.

[0045] The pose-optimized object-level 3D Mesh model is embedded into the original 3D scene. Mesh grounding processing is performed by combining the normal vector information of the ground point cloud. By adjusting the consistency of the normal vector direction between the bottom point set of the object-level 3D Mesh model and the ground point cloud, the physical alignment of the object-level 3D Mesh model with the ground region in the background point cloud data is completed. The non-rigid pose optimization method refers to constructing a multi-objective optimization function that includes point cloud registration error, physical contact constraint, and semantic consistency constraint to address the pose error of the object-level 3D Mesh model during the fusion with the background point cloud data. An optimization strategy based on energy minimization is adopted. The rotation matrix and translation vector of the Mesh model are iteratively adjusted. In each optimization iteration, the correspondence between the target Mesh and the background point cloud is calculated. The spatial pose and deformation coefficient of the model are updated by combining rigid transformation and local non-rigid deformation. Finally, the target Mesh pose that matches the background point cloud in terms of geometric structure, physical contact, and semantic layout is obtained.

[0046] The background point cloud data is subjected to structural simplification processing. The random sampling consistency plane fitting method is used to extract the background plane with regular structure in the scene. The background plane includes roads and walls. Outliers and redundant noise are removed, and key geometric structural features are retained.

[0047] The pose-aligned object-level 3D Mesh model is fused with the simplified background point cloud data to generate a complete 3D scene.

[0048] The beneficial effects of this invention are:

[0049] This invention proposes a 3D scene reconstruction method based on multimodal sensors of intelligent LED streetlights, significantly solving the reconstruction problems of existing technologies in dynamic scenes and large-scale urban environments. By combining LiDAR and high-resolution RGB images, and relying on the stable deployment of intelligent LED streetlights, continuous and stable multi-view data acquisition can be achieved, providing 360-degree real-time reconstruction capabilities for urban street scenes without blind spots. By accurately calibrating and projecting point cloud data with RGB images, this invention effectively solves the problem of insufficient accuracy in the fusion of point cloud and image data in existing technologies, improving geometric accuracy and texture details, thereby enabling the reconstruction of more realistic and detailed 3D models.

[0050] This invention also represents a breakthrough in dynamic target reconstruction. Traditional 3D reconstruction techniques often suffer from geometric distortion and blurring due to changes in viewpoint, occlusion, or object movement in dynamic environments, failing to guarantee reconstruction quality. However, this invention, through an improved point cloud segmentation method, an N-ICP non-rigid registration algorithm, and a generative model based on implicit representation, effectively addresses data loss or geometric discontinuities caused by occlusion and dynamic objects. This ensures high-precision 3D reconstruction results even in complex dynamic environments, especially in urban street scenes with dense traffic and complex building structures.

[0051] Furthermore, this invention employs advanced pose optimization and semantic constraint methods to effectively correct the spatial position and orientation of the target object within the scene, and ensures precise alignment between the target object and the background scene through physical contact constraints. Compared to traditional methods, this invention not only significantly improves computational accuracy and reconstruction results but also greatly enhances processing efficiency, enabling real-time data processing in city-level scenes and meeting the needs of applications such as smart cities and autonomous driving. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a flowchart of a three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light proposed in this invention;

[0054] Figure 2 This is a schematic diagram of a three-dimensional scene for a three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light proposed in this invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0056] refer to Figure 1-2 A method for reconstructing a 3D scene based on a multimodal sensor of an intelligent LED street light includes the following steps:

[0057] Acquire RGB images, perform preprocessing, and generate standardized RGB images;

[0058] The standardized RGB image is input into the visual feature encoding module to extract the two-dimensional bounding box information of the candidate object, and guide the object segmentation module to generate the two-dimensional segmentation mask of the corresponding target object.

[0059] Point cloud data is acquired, projected onto the pixel coordinate system of a standardized RGB image using a rigid extrinsic matrix and a camera intrinsic matrix, and target points are filtered using a two-dimensional segmentation mask to generate preliminary point cloud segmentation results.

[0060] Based on the parallax information of multiple LiDAR sensors, the preliminary point cloud segmentation results are completed at the object level to generate a complete point cloud segmentation result. The three-dimensional spatial coordinate information of each target point in the complete point cloud segmentation result is mapped to a standardized RGB image, and the corresponding pixel region is extracted to generate an object-level RGB image.

[0061] Using object-level RGB images and complete point cloud segmentation results as input, a generative model based on implicit representation is used for joint encoding, implicit representation and neural decoding to generate object-level 3D Mesh results.

[0062] The object-level mesh results are fused into the original 3D scene. By aligning the position and orientation through point cloud registration, pose optimization, and semantic constraints, the spatial reconstruction of the target object in the 3D scene is completed, generating a complete 3D scene.

[0063] This invention provides a 3D scene reconstruction method based on a multimodal sensor of intelligent LED streetlights. Employing multi-source data fusion technology, combining RGB images and LiDAR point cloud data, it effectively improves the accuracy and efficiency of 3D modeling. Through precise projection of standardized RGB images and point cloud data, and guided by object segmentation masks, this invention can accurately select target points, avoiding errors caused by data redundancy or occlusion issues in traditional methods. Furthermore, the use of a generative model based on implicit representation for joint encoding and decoding ensures high fidelity in both geometry and texture of the generated 3D mesh model, effectively improving reconstruction quality in dynamic scenes, especially in complex urban environments.

[0064] In this embodiment, the preprocessing includes normalizing pixel values ​​and adjusting the image size to a preset input resolution.

[0065] This invention performs image normalization and resizing operations during the preprocessing stage to ensure that the input image meets the requirements of subsequent processing. By adjusting the image size to a preset input resolution, computational efficiency and quality are guaranteed. Particularly in large-scale scene reconstruction, this effectively improves image processing speed and reduces computational burden. This image preprocessing method not only enhances the model's processing efficiency but also ensures accuracy in subsequent image feature extraction and target segmentation processes. It further optimizes the registration and fusion of images and point cloud data, resulting in a more accurate and reliable final 3D model.

[0066] In this embodiment, the generation of the two-dimensional segmentation mask specifically includes:

[0067] The standardized RGB image is input into the visual feature encoding module to extract multi-scale image feature representations containing spatial semantic information. The visual feature encoding module adopts a multi-scale hierarchical structure based on visual Transformer. By inputting the standardized RGB image, it generates a multi-scale image feature map containing global semantics and local details.

[0068] The multi-scale feature representation of the image is input into the object detection module, which outputs the category label and corresponding two-dimensional bounding box information for each object. The object detection module is built based on the pre-trained object detection Transformer architecture and uses a multi-head self-attention mechanism to perform semantic parsing of potential objects in the image. The two-dimensional bounding box information consists of a quadruple of center x-coordinate, center y-coordinate, bounding box width, and bounding box height in the image coordinate system.

[0069] The two-dimensional bounding box information is input into the object segmentation module, which guides the object segmentation module to generate a two-dimensional segmentation mask for the corresponding target object within each bounding box region. The object segmentation module is built based on the Segment Anything model, and the two-dimensional segmentation mask is a binary image with the same resolution as the normalized RGB image, used to indicate the pixel position of the target object.

[0070] This invention, by introducing a visual Transformer network and a pre-trained object detection module, effectively extracts multi-scale spatial semantic information and local texture features during image encoding, further enhancing the ability to identify potential targets in images. Based on these multi-scale features, this invention can accurately locate and segment target objects, ensuring accurate object identification and segmentation even in high-density or complex backgrounds. Through this image encoding-based processing method, this invention improves the accuracy and stability of object segmentation results, making it particularly suitable for target object extraction in dynamic scenes, ensuring high precision and efficiency in subsequent 3D reconstruction.

[0071] In this embodiment, the generation of the preliminary point cloud segmentation result specifically includes:

[0072] Point cloud data is acquired, and the spatial coordinate information of all three-dimensional points is extracted. The point cloud data is collected by a LiDAR sensor installed on a smart LED street light. The spatial coordinate information includes three parts: horizontal coordinate, vertical coordinate, and height coordinate.

[0073] Based on the extrinsic calibration results, the spatial coordinate information of each three-dimensional point in the point cloud data is projected onto the two-dimensional pixel coordinate system of the standardized RGB image using the rigid extrinsic matrix between the camera and LiDAR and the camera intrinsic matrix, so as to obtain the corresponding pixel coordinate information.

[0074] The external parameter calibration result refers to the rigid transformation matrix that describes the three-dimensional spatial geometric relationship between the camera and LiDAR, which is calculated based on the common reference coordinate system of the two sensors after the camera and LiDAR on the smart LED street light respectively collect the image and point cloud data of the same checkerboard. The camera's attitude information relative to the checkerboard is calculated and extracted using the Zhang Zhengyou calibration method, and the LiDAR's attitude information relative to the checkerboard is extracted using the point cloud plane fitting method.

[0075] The rigid extrinsic parameter matrix is ​​the transformation matrix of the three-dimensional spatial geometric relationship between the camera and the LiDAR, and it consists of a rotation matrix and a translation vector.

[0076] The camera intrinsic parameter matrix is ​​a set of intrinsic parameters of the camera imaging model, specifically including focal length, principal point coordinates, and pixel scaling factor;

[0077] The pose parameters include a rotation matrix and a translation vector;

[0078] The pixel coordinates of each 3D point are judged. If the pixel value at the corresponding position in the 2D segmentation mask is a foreground value, it is determined to be a target point. If the value is a background value, it is discarded. The foreground value is the pixel value of the pixel region of the target object in the 2D segmentation mask, which is usually set to 1. The background value is the pixel value of the pixel region of the non-target region in the 2D segmentation mask, which is usually set to 0.

[0079] By combining all the 3D points identified as target points, a preliminary point cloud segmentation result constituting the target object is generated.

[0080] This invention combines parallax information from multiple LiDAR sensors to perform object-level completion on the initial point cloud segmentation results, effectively solving the problem of missing point cloud data caused by differences in sensor viewpoints and occlusion. Through precise 3D spatial coordinate mapping and point cloud data completion processing, this invention achieves high-precision reconstruction of complex object shapes and backgrounds in urban environments. Furthermore, by generating object-level RGB images, this invention provides ample texture support for subsequent 3D mesh model generation, resulting in a reconstructed 3D scene that not only possesses high-precision geometric information but also realistic texture effects, enhancing the realism and application value of the reconstruction results.

[0081] In this embodiment, the generation of the object-level RGB image specifically includes:

[0082] Perform normal vector analysis and curvature estimation on the three-dimensional spatial coordinate information in the preliminary point cloud segmentation results, detect edge regions with abrupt changes in normal vectors or a sudden drop in point density, and generate missing regions.

[0083] Point cloud data collected by multiple LiDAR sensors is extracted to generate multi-view point cloud data. Based on the rigid extrinsic matrix in the extrinsic calibration results, the multi-view point cloud data is uniformly transformed to the common three-dimensional coordinate system of the target object.

[0084] Based on multi-view point cloud data, the depth difference between each view is calculated and used as a constraint. Combined with the N-ICP registration method, the multi-view point cloud data is aligned to the same spatial coordinate system. During the registration process, the matching relationship between point clouds is optimized through multiple rounds of iteration to solve the geometric discontinuity problem caused by occlusion or view difference. For missing regions, interpolation methods are used to fill in the missing regions. The point cloud data of the missing regions is inferred using the geometric features and depth information of the surrounding point clouds. Finally, fused multi-view point cloud data is generated.

[0085] The N-ICP registration method is an improved point cloud registration algorithm designed to address the limitations of traditional ICP methods in handling non-rigid deformation or occlusion problems. Unlike traditional rigid ICP algorithms, the N-ICP method introduces a deformation model, allowing point clouds to undergo local deformation during registration, thus better adapting to shape changes in the scene. Specifically, the N-ICP algorithm finds the closest point pair between two point clouds through nearest neighbor search and calculates the transformation relationship. Unlike traditional methods, N-ICP minimizes the objective function and iteratively optimizes the transformation parameters through multiple rounds, minimizing the matching error between the two sets of point clouds in space. The N-ICP algorithm not only focuses on the geometric matching relationship of point clouds but also introduces deformation constraints, which can effectively handle geometric discontinuities caused by object deformation or occlusion, thereby improving registration accuracy and robustness.

[0086] The deformation model is a mathematical model used to describe the geometric changes of an object or point cloud during the deformation process. In the point cloud registration process, by introducing local degrees of freedom on the basis of rigid transformation, non-rigid deformations such as local stretching, compression or bending are performed on the point cloud to adapt to the inconsistencies caused by occlusion, viewing angle differences or local shape changes on the object surface, thereby improving the flexibility and accuracy of point cloud registration.

[0087] Based on the fusion of multi-view point cloud data, local surface fitting processing is performed on the boundary region to improve the overall geometric continuity and boundary smoothness of the point cloud, and output the complete segmentation result of the point cloud. The boundary region is the location where there is still geometric discontinuity in the fusion of multi-view point cloud data.

[0088] Based on the spatial coordinate information of each target point in the complete point cloud segmentation result, and combined with the rigid extrinsic matrix between the camera and LiDAR and the camera intrinsic matrix, the three-dimensional spatial coordinate information of the target point is projected onto the two-dimensional pixel coordinate system of the standardized RGB image, the corresponding pixel region is extracted, and an object-level RGB image is generated.

[0089] This invention employs a generative model based on implicit representation. By jointly encoding and decoding point cloud segmentation results and RGB images, it successfully fuses point cloud data with image texture information to generate a high-quality object-level 3D mesh model. Through implicit field modeling, the geometric shape and texture information of objects can be accurately represented, further improving the structural integrity and texture fidelity of the 3D scene. This method not only ensures high accuracy and high fidelity of 3D modeling results in complex urban environments but also effectively handles the reconstruction of dynamic targets, providing a more stable and efficient 3D reconstruction solution.

[0090] In this embodiment, the generation of the object-level Mesh result specifically includes:

[0091] The complete point cloud segmentation result is input into the point cloud encoding module of the generative model based on implicit representation. Spatial structural features are extracted using a sparse 3D convolutional network to generate implicit geometric representation vectors.

[0092] The implicit representation-based generative model refers to a neural network model that combines point cloud and image information for 3D mesh reconstruction. Based on the original 3DShape2VecSet method, an image-guided multimodal coding structure is designed. By encoding image features into spatial embedding vectors and fusing them with the implicit representation of point clouds, the joint expression of texture and geometric information is achieved. Specifically, it includes a point cloud coding module, an image coding module, a feature fusion module, an implicit field modeling module, and a texture mapping module.

[0093] The point cloud encoding module is composed of a sparse three-dimensional convolutional neural network, which uses sparse voxel representation to perform hierarchical encoding on object-level point cloud data, converting the original point cloud into an implicit geometric representation vector with local geometric perception capability.

[0094] The image encoding module is composed of a visual Transformer network, which extracts global semantic information and local texture features of the body-level RGB image through a multi-scale encoder, and encodes the features into image guided representation vectors through a spatial embedding mechanism.

[0095] The feature fusion module consists of a cross-modal attention mechanism, which guides and enhances the implicit geometric representation vector through image-guided representation vectors, thereby achieving deep fusion of texture information and geometric structure in the joint feature space.

[0096] The implicit field modeling module consists of a coordinate-aware neural network and a multilayer perceptron structure. Based on continuous spatial coordinate input and joint feature representation, it predicts the occupancy probability of each three-dimensional spatial sampling point in the implicit space and constructs a continuous implicit field function.

[0097] The coordinate-aware neural network is a neural network structure that encodes three-dimensional spatial coordinates, aiming to improve the network's ability to perceive spatial positions. By receiving continuous three-dimensional coordinate information and combining it with other input features, it learns the correlation between spatial coordinates and the geometric structure of the target object, and can more accurately predict the occupancy probability of each sampling point in three-dimensional space, thereby effectively constructing an implicit field function.

[0098] The texture mapping module consists of a spatial alignment network and a texture decoding branch. Based on the spatial mapping relationship between the vertices of the triangular mesh and the image pixels, it accurately maps the texture information in the image guiding representation vector to the surface of the triangular mesh, generating a three-dimensional Mesh model with high-fidelity texture details.

[0099] The three-dimensional spatial sampling point refers to a virtual coordinate point obtained by uniform or structural sampling within the three-dimensional space of the object;

[0100] The object-level RGB image is input into the image encoding module to extract its multi-scale semantic information and local texture features, and then encoded into an image guided representation vector through a spatial embedding mechanism.

[0101] The image guided representation vector and implicit geometric representation vector are input into the feature fusion module. The two are weighted and calculated through a cross-modal attention mechanism to automatically learn the relationship between image texture information and geometric information. By calculating similarity and difference, key features are extracted and a unified multimodal joint feature representation is generated.

[0102] The multimodal joint feature representation is input into the implicit field modeling module. Based on the continuous spatial coordinate query mechanism, the neural decoding network predicts the spatial occupancy probability of each three-dimensional spatial sampling point. The continuous spatial coordinate query mechanism refers to the process of implicit field modeling in which any continuous coordinate point in three-dimensional space is used as input, and the joint feature representation vector obtained by multimodal fusion is used to guide the neural decoding network to predict the spatial occupancy status of each continuous coordinate point. Based on the coordinate perception neural network, the three-dimensional spatial sampling point is concatenated with the multimodal feature vector and input into the multilayer perceptron. The coordinate position is calculated point by point to determine whether it is on the surface of the object. In this way, a continuously defined implicit field function is constructed in the entire three-dimensional space, realizing fine three-dimensional structure expression and high-resolution mesh extraction.

[0103] An implicit field function is constructed based on the spatial occupancy probability. Triangular patches are extracted using the differentiable Marching Cubes method to generate a structurally continuous triangular mesh model. The differentiable Marching Cubes method refers to dividing the constructed continuous implicit field function into voxels based on the implicit field modeling, and judging its topological structure in each voxel cell according to the spatial occupancy probability of the sampling point, thereby extracting isosurfaces to form a triangular mesh.

[0104] The image-guided representation vector is input into the texture mapping module. Based on the spatial projection relationship between the triangular mesh and the image, the image texture is mapped onto the surface of the triangular mesh to generate an object-level 3D Mesh result with texture information.

[0105] This invention, based on the generation of a 3D mesh model, utilizes non-rigid pose optimization methods and semantic constraints to optimize the model's pose. This enables precise adjustment of the target object's spatial position and orientation within the 3D scene, ensuring geometric consistency between the target object and the background scene. Through multiple rounds of optimization iterations, this invention effectively solves pose deviation problems caused by sensor errors, occlusion, or viewpoint differences, ultimately achieving perfect alignment between the target object and the background point cloud. This optimization process significantly improves the accuracy and reconstruction effect of the final 3D scene, making it particularly suitable for dynamic targets and multi-view data processing in complex urban environments, and possessing broad application prospects.

[0106] In this embodiment, the generation of the complete 3D scene specifically includes:

[0107] Extracting the object-level 3D Mesh result and the background point cloud data in the original 3D scene, performing initial coarse registration of the object-level 3D Mesh model based on the spatial geometric distribution relationship, and calculating the initial pose parameters in the original 3D scene. The initial pose parameters include rotation matrix and translation vector. The background point cloud data in the original 3D scene refers to the set of 3D spatial points that were not identified as target points during the initial segmentation and completion of the point cloud.

[0108] Point cloud registration error, physical contact constraint, and semantic consistency constraint are used as loss function terms to construct a pose optimization objective function. The spatial position and pose of the object-level 3D Mesh model are optimized by minimizing the pose optimization objective function. The point cloud registration error term refers to the geometric deviation between the object-level 3D Mesh model and the background point cloud data in 3D space. The physical contact constraint term refers to the distance error between each contact point pair between the bottom point set of the Mesh model and the ground area in the background point cloud data, which quantifies the contact relationship and is ultimately used as part of the optimization objective function. The semantic consistency constraint term is formed by calculating the rotation error and position error of the target object and then weighting and summing them to form the loss function term.

[0109] The non-rigid pose optimization method is used to iteratively adjust the pose parameters of the object-level 3D Mesh model. By performing multiple rounds of optimization and iteration on the pose optimization objective function, the rotation matrix and translation vector are updated to generate the pose-optimized object-level 3D Mesh model.

[0110] The pose-optimized object-level 3D Mesh model is embedded into the original 3D scene. Mesh grounding processing is performed by combining the normal vector information of the ground point cloud. By adjusting the consistency of the normal vector direction between the bottom point set of the object-level 3D Mesh model and the ground point cloud, the physical alignment of the object-level 3D Mesh model with the ground region in the background point cloud data is completed. The non-rigid pose optimization method refers to constructing a multi-objective optimization function that includes point cloud registration error, physical contact constraint, and semantic consistency constraint to address the pose error of the object-level 3D Mesh model during the fusion with the background point cloud data. An optimization strategy based on energy minimization is adopted. The rotation matrix and translation vector of the Mesh model are iteratively adjusted. In each optimization iteration, the correspondence between the target Mesh and the background point cloud is calculated. The spatial pose and deformation coefficient of the model are updated by combining rigid transformation and local non-rigid deformation. Finally, the target Mesh pose that matches the background point cloud in terms of geometric structure, physical contact, and semantic layout is obtained.

[0111] The background point cloud data is subjected to structural simplification processing. The random sampling consistency plane fitting method is used to extract the background plane with regular structure in the scene. The background plane includes roads and walls. Outliers and redundant noise are removed, and key geometric structural features are retained.

[0112] The pose-aligned object-level 3D Mesh model is fused with the simplified background point cloud data to generate a complete 3D scene with continuous boundaries and faithful texture.

[0113] This invention successfully solves the registration accuracy problem of traditional point cloud registration algorithms in complex scenes by combining depth consistency constraints based on multi-view point cloud data with the Non-rigid Iterative Nearest Point Registration (N-ICP) method. By calculating the depth difference between each viewpoint, the N-ICP algorithm efficiently aligns the multi-view point cloud data, eliminating geometric discontinuities caused by occlusion and viewpoint differences. Through this multi-view point cloud data fusion and missing region completion technique, this invention can generate high-precision and complete point cloud data, further improving the quality and accuracy of 3D reconstruction, especially showing significant advantages in processing large-scale urban scenes.

[0114] Example 1:

[0115] To verify the feasibility of this invention in practice, it was applied to a major road intersection area in a large city. This area is characterized by dense traffic, complex pedestrian flow, and a wide variety of building structures, traffic signs, green belts, and other elements, making it a typical high-dynamic, structurally diverse, and textured open urban environment. Multiple sets of intelligent LED streetlights have been deployed in this area. Each set of streetlights is fixedly installed on both sides of the road with uniform spacing and is equipped with a high-resolution RGB camera and a solid-state LiDAR sensor, enabling continuous acquisition of images and point cloud data, naturally forming a multimodal sensing network.

[0116] In practical applications, the 3D scene reconstruction method of this invention relies on the stable deployment and power supply and communication capabilities of intelligent LED streetlights to achieve efficient modeling of dynamic targets (such as vehicles, pedestrians, bicycles, etc.) in urban street scenes. The specific process is as follows: First, RGB images are acquired and normalized and resized to generate a unified standardized image input. Next, multi-scale image features are extracted through a visual Transformer-based encoding module. Combined with a pre-trained detection network and segmentation model, the bounding box localization and segmentation mask generation of target objects in the street scene are achieved, thereby effectively identifying the foreground region that needs to be modeled.

[0117] Subsequently, using the extrinsic calibration mechanism proposed in this invention, multi-source point cloud data is accurately projected onto the image coordinate system, achieving spatial correspondence between point clouds and images. Guided by the segmentation mask, only point cloud data corresponding to the target foreground region is retained, significantly reducing redundant computational burden and improving the accuracy of point cloud segmentation. Considering potential issues such as sensor occlusion and missing object parts in urban scenes, this invention further combines point cloud information from multiple streetlight perspectives, utilizing non-rigid registration and surface fitting techniques to complete object contours, restore missing geometric structures, and construct object-level RGB images based on projection relationships, ensuring complete texture support for subsequent modeling.

[0118] In the mesh generation stage, this invention, based on an improved implicit modeling method, jointly encodes object-level point clouds and image features, introduces a cross-modal feature fusion strategy, utilizes a continuous spatial query mechanism combined with a neural decoder to predict spatial occupancy relationships, and completes 3D model reconstruction through a differentiable triangular mesh extraction algorithm. Finally, leveraging the geometric relationship between the 3D mesh model and the background point cloud, a non-rigid pose optimization method is used to further correct the position and orientation of the target object in the scene, achieving physical contact alignment with the ground.

[0119] In real-world road tests, this method can handle dynamic targets such as fast-moving vehicles, temporarily parked delivery riders, and pedestrians moving on the street, achieving continuous and uninterrupted high-quality 3D reconstruction. The application does not rely on manual annotation or require high-precision GPS or IMU support; high-precision data acquisition and modeling can be completed solely through the vision and LiDAR sensors built into the LED streetlights themselves.

[0120] Table 1. Performance Comparison of the Invention Method and Traditional 3D Reconstruction Methods

[0121] Comparison indicators Method of the present invention Traditional methods Point cloud reconstruction accuracy (cm) 1.2 3.6 Mesh model accuracy (cm) 1.5 4.3 Texture reproduction error (decibels) 34.8 27.5 Reconstruction integrity rate (%) under occlusion conditions 91.2 68.1 Inter-frame consistency error in dynamic scenes (cm) 0.9 3.0 City-level scene processing efficiency (frames / minute) 33 10 Segmentation accuracy IoU (%) 89.6 66.7

[0122] As can be clearly seen from Table 1, the method of the present invention is superior to the traditional method in many core technical indicators of 3D scene reconstruction.

[0123] In terms of point cloud reconstruction and mesh modeling accuracy, this invention achieves 1.2 cm and 1.5 cm respectively, while traditional methods achieve 3.6 cm and 4.3 cm, representing an accuracy improvement of more than 2 times. This is mainly due to the multi-view point cloud completion method proposed in this invention. Based on the sparse multi-view LiDAR network deployed by intelligent LED streetlights, and combined with normal vector analysis and non-rigid registration algorithms, it effectively completes and interpolates sparse regions of the point cloud, significantly improving the structural reconstruction accuracy of the target object.

[0124] In terms of texture restoration, the PSNR of the method in this invention reaches 34.8 dB, which is much higher than the 27.5 dB of the traditional method, indicating a significant improvement in texture clarity and detail restoration capability. This performance improvement stems from the multimodal generation model based on implicit representation in this invention, which innovatively introduces image-guided representation vectors and deeply fuses image texture with point cloud geometric information through a cross-modal attention mechanism to achieve high-fidelity texture mapping.

[0125] Regarding robustness in occlusion and dynamic scenes, this invention achieves a reconstruction integrity rate of 91.2% in occluded environments, compared to only 68.1% for traditional methods. Furthermore, the inter-frame consistency error in dynamic scenes is only 0.9 cm, a significant improvement over the traditional 3.0 cm. This advantage stems from the invention's combination of semantic segmentation and multi-view consistency judgment strategies, effectively suppressing reconstruction gaps and mismatches in occlusion and motion.

[0126] In terms of efficiency, the processing efficiency for city-level scenes reaches 33 frames per minute, which is more than three times that of traditional methods. This is thanks to the stable deployment of LED street light edge nodes and the collaborative preprocessing mechanism of images and point clouds, which allows a large amount of computation to be completed at the front-end nodes, significantly improving the throughput of the overall reconstruction process.

[0127] Finally, in terms of semantic segmentation accuracy, the method of this invention achieves an IoU of 89.6%, which is nearly 23% higher than traditional methods. The combination of the Segment Anything model and the visual Transformer structure makes the segmentation results more accurate, providing a more stable foundation for subsequent point cloud selection and mesh generation.

[0128] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for three-dimensional scene reconstruction based on a multimodal sensor of an intelligent LED street light, characterized in that, Includes the following steps: Acquire RGB images, perform preprocessing, and generate standardized RGB images; The standardized RGB image is input into the visual feature encoding module to extract the two-dimensional bounding box information of the candidate object, and guide the object segmentation module to generate the two-dimensional segmentation mask of the corresponding target object. Point cloud data is acquired, projected onto the pixel coordinate system of a standardized RGB image using a rigid extrinsic matrix and a camera intrinsic matrix, and target points are filtered using a two-dimensional segmentation mask to generate preliminary point cloud segmentation results. Based on the parallax information of multiple LiDAR sensors, the preliminary point cloud segmentation results are completed at the object level to generate a complete point cloud segmentation result. The three-dimensional spatial coordinate information of each target point in the complete point cloud segmentation result is mapped to a standardized RGB image, and the corresponding pixel region is extracted to generate an object-level RGB image. Using object-level RGB images and complete point cloud segmentation results as input, a generative model based on implicit representation is used for joint encoding, implicit representation and neural decoding to generate object-level 3D Mesh results. The object-level mesh results are fused into the original 3D scene. By aligning the position and orientation through point cloud registration, pose optimization, and semantic constraints, the spatial reconstruction of the target object in the 3D scene is completed, generating a complete 3D scene.

2. The three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light according to claim 1, characterized in that, The preprocessing includes normalizing pixel values ​​and adjusting the image size to a preset input resolution.

3. The three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light according to claim 1, characterized in that, The generation of the two-dimensional segmentation mask specifically includes: The standardized RGB image is input into the visual feature encoding module to extract multi-scale image feature representations containing spatial semantic information. The visual feature encoding module adopts a multi-scale hierarchical structure based on visual Transformer. By inputting the standardized RGB image, it generates a multi-scale image feature map containing global semantics and local details. The multi-scale feature representation of the image is input into the object detection module, which outputs the category label and corresponding two-dimensional bounding box information for each object. The object detection module is built based on the pre-trained object detection Transformer architecture and uses a multi-head self-attention mechanism to perform semantic parsing of potential objects in the image. The two-dimensional bounding box information consists of a quadruple of center x-coordinate, center y-coordinate, bounding box width, and bounding box height in the image coordinate system. The two-dimensional bounding box information is input into the object segmentation module, which guides the object segmentation module to generate a two-dimensional segmentation mask for the corresponding target object within each bounding box region. The object segmentation module is constructed based on the Segment Anything model, and the two-dimensional segmentation mask is a binary image with the same resolution as the standardized RGB image.

4. The three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light according to claim 1, characterized in that, The generation of the preliminary point cloud segmentation result specifically includes: Point cloud data is acquired, and the spatial coordinate information of all three-dimensional points is extracted. The point cloud data is collected by a LiDAR sensor installed on a smart LED street light. The spatial coordinate information includes three parts: horizontal coordinate, vertical coordinate, and height coordinate. Based on the extrinsic calibration results, the spatial coordinate information of each three-dimensional point in the point cloud data is projected onto the two-dimensional pixel coordinate system of the standardized RGB image using the rigid extrinsic matrix between the camera and LiDAR and the camera intrinsic matrix, so as to obtain the corresponding pixel coordinate information. The external parameter calibration result refers to the rigid transformation matrix that describes the three-dimensional spatial geometric relationship between the camera and LiDAR, which is calculated based on the common reference coordinate system of the two sensors after the camera and LiDAR on the smart LED street light respectively collect the image and point cloud data of the same checkerboard. The camera's attitude information relative to the checkerboard is calculated and extracted using the Zhang Zhengyou calibration method, and the LiDAR's attitude information relative to the checkerboard is extracted using the point cloud plane fitting method. The rigid extrinsic parameter matrix is ​​the transformation matrix of the three-dimensional spatial geometric relationship between the camera and the LiDAR, and it consists of a rotation matrix and a translation vector. The camera intrinsic parameter matrix is ​​a set of intrinsic parameters of the camera imaging model, specifically including focal length, principal point coordinates, and pixel scaling factor; The pose parameters include a rotation matrix and a translation vector; The pixel coordinates of each 3D point are judged. If the pixel value at the corresponding position in the 2D segmentation mask is a foreground value, it is determined to be a target point. If the value is a background value, it is discarded. The foreground value is the pixel value of the pixel region of the target object in the 2D segmentation mask, which is usually set to 1. The background value is the pixel value of the pixel region of the non-target region in the 2D segmentation mask, which is usually set to 0. By combining all the 3D points identified as target points, a preliminary point cloud segmentation result constituting the target object is generated.

5. A three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light according to claim 1, characterized in that, The generation of the object-level RGB image specifically includes: Perform normal vector analysis and curvature estimation on the three-dimensional spatial coordinate information in the preliminary point cloud segmentation results, detect edge regions with abrupt changes in normal vectors or a sudden drop in point density, and generate missing regions. Point cloud data collected by multiple LiDAR sensors is extracted to generate multi-view point cloud data. Based on the rigid extrinsic matrix in the extrinsic calibration results, the multi-view point cloud data is uniformly transformed to the common three-dimensional coordinate system of the target object. Based on multi-view point cloud data, the depth difference between each view is calculated and used as a constraint. Combined with the N-ICP registration method, the multi-view point cloud data is aligned to the same spatial coordinate system. During the registration process, the matching relationship between point clouds is optimized through multiple rounds of iteration to solve the geometric discontinuity problem caused by occlusion or view difference. For missing regions, interpolation methods are used to fill in the missing regions. The point cloud data of the missing regions is inferred using the geometric features and depth information of the surrounding point clouds. Finally, fused multi-view point cloud data is generated. Based on the fusion of multi-view point cloud data, local surface fitting processing is performed on the boundary region to improve the overall geometric continuity and boundary smoothness of the point cloud, and output the complete segmentation result of the point cloud. The boundary region is the location where there is still geometric discontinuity in the fusion of multi-view point cloud data. Based on the spatial coordinate information of each target point in the complete point cloud segmentation result, and combined with the rigid extrinsic matrix between the camera and LiDAR and the camera intrinsic matrix, the three-dimensional spatial coordinate information of the target point is projected onto the two-dimensional pixel coordinate system of the standardized RGB image, the corresponding pixel region is extracted, and an object-level RGB image is generated.

6. The three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light according to claim 1, characterized in that, The generation of the object-level Mesh result specifically includes: The complete point cloud segmentation result is input into the point cloud encoding module of the generative model based on implicit representation. Spatial structural features are extracted using a sparse 3D convolutional network to generate implicit geometric representation vectors. The implicit representation-based generative model refers to a neural network model that combines point cloud and image information for 3D mesh reconstruction. Based on the original 3DShape2VecSet method, an image-guided multimodal coding structure is designed. By encoding image features into spatial embedding vectors and fusing them with the implicit representation of point clouds, the joint expression of texture and geometric information is achieved. Specifically, it includes a point cloud coding module, an image coding module, a feature fusion module, an implicit field modeling module, and a texture mapping module. The object-level RGB image is input into the image encoding module to extract its multi-scale semantic information and local texture features, and then encoded into an image guided representation vector through a spatial embedding mechanism. The image guided representation vector and implicit geometric representation vector are input into the feature fusion module. The two are weighted and calculated through a cross-modal attention mechanism to automatically learn the relationship between image texture information and geometric information. By calculating similarity and difference, key features are extracted and a unified multimodal joint feature representation is generated. The multimodal joint feature representation is input into the implicit field modeling module. Based on the continuous spatial coordinate query mechanism, the neural decoding network predicts the spatial occupancy probability of each three-dimensional spatial sampling point. The continuous spatial coordinate query mechanism refers to the process of implicit field modeling in which any continuous coordinate point in three-dimensional space is used as input, and the joint feature representation vector obtained by multimodal fusion is used to guide the neural decoding network to predict the spatial occupancy status of each continuous coordinate point. Based on the coordinate perception neural network, the three-dimensional spatial sampling point is concatenated with the multimodal feature vector and input into the multilayer perceptron. The coordinate position is calculated point by point to determine whether it is on the surface of the object. Thus, a continuously defined implicit field function is constructed in the entire three-dimensional space to realize Mesh extraction. An implicit field function is constructed based on the spatial occupancy probability. Triangular patches are extracted using the differentiable Marching Cubes method to generate a structurally continuous triangular mesh model. The differentiable Marching Cubes method refers to dividing the constructed continuous implicit field function into voxels based on the implicit field modeling, and judging its topological structure in each voxel cell according to the spatial occupancy probability of the sampling point, thereby extracting isosurfaces to form a triangular mesh. The image-guided representation vector is input into the texture mapping module. Based on the spatial projection relationship between the triangular mesh and the image, the image texture is mapped onto the surface of the triangular mesh to generate an object-level 3D Mesh result with texture information.

7. A three-dimensional scene reconstruction method based on a multimodal sensor of an intelligent LED street light according to claim 1, characterized in that, The generation of the complete 3D scene specifically includes: Extracting the object-level 3D Mesh result and the background point cloud data in the original 3D scene, performing initial coarse registration of the object-level 3D Mesh model based on the spatial geometric distribution relationship, and calculating the initial pose parameters in the original 3D scene. The initial pose parameters include rotation matrix and translation vector. The background point cloud data in the original 3D scene refers to the set of 3D spatial points that were not identified as target points during the initial segmentation and completion of the point cloud. Point cloud registration error, physical contact constraint, and semantic consistency constraint are used as loss function terms to construct a pose optimization objective function. The spatial position and pose of the object-level 3D Mesh model are optimized by minimizing the pose optimization objective function. The point cloud registration error term refers to the geometric deviation between the object-level 3D Mesh model and the background point cloud data in 3D space. The physical contact constraint term refers to the distance error between each contact point pair between the bottom point set of the Mesh model and the ground area in the background point cloud data, which quantifies the contact relationship and is ultimately used as part of the optimization objective function. The semantic consistency constraint term is formed by calculating the rotation error and position error of the target object and then weighting and summing them to form the loss function term. The non-rigid pose optimization method is used to iteratively adjust the pose parameters of the object-level 3D Mesh model. By performing multiple rounds of optimization and iteration on the pose optimization objective function, the rotation matrix and translation vector are updated to generate the pose-optimized object-level 3D Mesh model. The pose-optimized object-level 3D Mesh model is embedded into the original 3D scene. Mesh grounding processing is performed by combining the normal vector information of the ground point cloud. By adjusting the consistency of the normal vector direction between the bottom point set of the object-level 3D Mesh model and the ground point cloud, the physical alignment of the object-level 3D Mesh model with the ground region in the background point cloud data is completed. The non-rigid pose optimization method refers to constructing a multi-objective optimization function that includes point cloud registration error, physical contact constraint, and semantic consistency constraint to address the pose error of the object-level 3D Mesh model during the fusion with the background point cloud data. An optimization strategy based on energy minimization is adopted. The rotation matrix and translation vector of the Mesh model are iteratively adjusted. In each optimization iteration, the correspondence between the target Mesh and the background point cloud is calculated. The spatial pose and deformation coefficient of the model are updated by combining rigid transformation and local non-rigid deformation. Finally, the target Mesh pose that matches the background point cloud in terms of geometric structure, physical contact, and semantic layout is obtained. The background point cloud data is subjected to structural simplification processing. The random sampling consistency plane fitting method is used to extract the background plane with regular structure in the scene. The background plane includes roads and walls. Outliers and redundant noise are removed, and key geometric structural features are retained. The pose-aligned object-level 3D Mesh model is fused with the simplified background point cloud data to generate a complete 3D scene.

Images