A method, system, device, and medium for 3D reconstruction based on monocular video.

Abstract

This application discloses a method, system, device, and medium for 3D reconstruction based on monocular video. The method includes: acquiring and processing data of an indoor space using a monocular camera to obtain monocular video; performing sliding window segmentation on the monocular video according to the spatial volume of the indoor space to obtain video segments; performing local reconstruction processing on the video segments to obtain locally reconstructed point clouds; performing keyframe joint registration processing on the locally reconstructed point clouds to obtain registered scene frames; and performing global scene optimization processing on the registered scene frames according to spatial constraints to obtain the 3D reconstruction result. The embodiments of this application can improve the accuracy of 3D reconstruction and can be widely applied in the field of computer vision technology.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, system, device and medium for three-dimensional reconstruction based on monocular video. Background Technology

[0002] In related technologies, 3D reconstruction methods typically rely on acquiring large amounts of 3D point cloud or image data using multiple stereo cameras or similar devices, and then converting this data into a 3D model using 3D reconstruction algorithms. However, in practical applications, it has been found that these methods suffer from several drawbacks in small spatial environments. The narrow field of view leads to an increase in repetitive feature regions, causing registration ambiguity. Furthermore, the limited range of motion reduces the effectiveness of motion parallax, exacerbating monocular scale drift and impacting the efficiency of 3D reconstruction. In summary, the technical problems existing in these technologies require improvement. Summary of the Invention

[0003] The main objective of this application is to propose a method, system, device, and medium for three-dimensional reconstruction based on monocular video, which can improve the accuracy of three-dimensional reconstruction.

[0004] To achieve the above objectives, one aspect of this application proposes a 3D reconstruction method based on monocular video, the method comprising:

[0005] The indoor space is collected and processed using a monocular camera to obtain monocular video.

[0006] The monocular video is segmented using a sliding window based on the spatial volume of the indoor space to obtain video segments.

[0007] The video segment is subjected to local reconstruction processing to obtain a locally reconstructed point cloud;

[0008] The locally reconstructed point cloud is subjected to keyframe joint registration processing to obtain the registered scene frame;

[0009] The registered scene frames are subjected to global scene optimization processing based on spatial constraints to obtain the 3D reconstruction results.

[0010] In some embodiments, the step of performing sliding window segmentation on the monocular video based on the spatial volume of the indoor space to obtain video segments includes the following steps:

[0011] The volume of the indoor space is calculated based on the monocular video.

[0012] The sliding window is initialized based on the spatial volume, and the length of the sliding window is adjusted based on motion blur detection to obtain the target sliding window;

[0013] The monocular video is segmented according to the target sliding window to obtain the video segment.

[0014] In some embodiments, the process of performing local reconstruction processing on the video segment to obtain a locally reconstructed point cloud includes the following steps:

[0015] The video segment is processed by an image encoder to extract features, and the extracted features are fused by a gated loop unit to obtain spatial scene features.

[0016] The spatial scene features are fused using a keyframe decoder to obtain multi-view information.

[0017] By supporting frame decoder to supplement keyframe information of the spatial scene features, keyframe information is obtained;

[0018] Based on spatial locality constraints, bidirectional cross-attention calculation is performed on the multi-view information and the keyframe information to obtain fused features;

[0019] The point cloud regression module based on deformable convolution performs regression prediction processing on the fused features to obtain the local reconstructed point cloud.

[0020] In some embodiments, the keyframe joint registration process of the locally reconstructed point cloud to obtain a registered scene frame includes the following steps:

[0021] Obtain the scene frame buffer pool, which includes historical scene frames;

[0022] The local reconstructed point cloud is subjected to coordinate transformation to obtain global point cloud data;

[0023] The global point cloud data is registered and retrieved based on the scene frame buffer pool to obtain the registered scene frame.

[0024] In some embodiments, the step of performing registration retrieval processing on the scene frame buffer pool based on the global point cloud data to obtain the registered scene frame includes the following steps:

[0025] Based on the global point cloud data, cosine similarity retrieval is performed on each historical scene frame in the scene frame buffer pool to generate a keyframe set.

[0026] Spatiotemporal feature alignment processing is performed on the keyframe set to obtain cross-keyframe spatiotemporal features;

[0027] Based on the cross-keyframe spatiotemporal features, the keyframe set is subjected to 3D point cloud registration processing to obtain a registered point cloud.

[0028] The registered point cloud is subjected to point cloud fusion processing to obtain the registered scene frame.

[0029] In some embodiments, the step of performing global scene optimization processing on the registered scene frame according to spatial constraints to obtain the 3D reconstruction result includes the following steps:

[0030] The registered scene frame is subjected to point cloud optimization processing to obtain point cloud optimization data;

[0031] The point cloud optimization data is subjected to planar constraint optimization processing to obtain planar optimization data;

[0032] The planar optimization data is subjected to spatial topology optimization processing to obtain the three-dimensional reconstruction result.

[0033] In some embodiments, performing spatial topology optimization processing on the planar optimization data to obtain the three-dimensional reconstruction result includes the following steps:

[0034] The planar optimization data is subjected to topology construction processing to obtain a scene topology map;

[0035] The scene topology graph is iteratively optimized using a graph convolutional network to obtain the 3D reconstruction result.

[0036] To achieve the above objectives, another aspect of this application proposes a 3D reconstruction system based on monocular video, the system comprising:

[0037] The first module is used to acquire and process video of the indoor space through a monocular camera to obtain monocular video.

[0038] The second module is used to perform sliding window segmentation on the monocular video according to the spatial volume of the indoor space to obtain video segments.

[0039] The third module is used to perform local reconstruction processing on the video segment to obtain a local reconstructed point cloud;

[0040] The fourth module is used to perform keyframe joint registration processing on the local reconstructed point cloud to obtain the registered scene frame;

[0041] The fifth module is used to perform global scene optimization processing on the registered scene frame according to spatial constraints to obtain the three-dimensional reconstruction result.

[0042] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0043] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0044] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, device, and medium for 3D reconstruction based on monocular video. This scheme obtains monocular video by acquiring and processing data from an indoor space using a monocular camera. Video segments are obtained by sliding window segmentation of the monocular video according to the spatial volume of the indoor space. The window length can be adaptively adjusted according to the spatial volume, increasing the local window overlap rate and reducing monocular scale drift, thus providing a data foundation for subsequent local reconstruction. Furthermore, this scheme obtains a locally reconstructed point cloud by performing local reconstruction processing on the video segments, obtains a registered scene frame by performing keyframe joint registration processing on the locally reconstructed point cloud, and obtains the 3D reconstruction result by performing global scene optimization processing on the registered scene frame according to spatial constraints. This allows for detection of reconstruction completeness based on spatial constraints, reducing registration errors and improving the accuracy of 3D reconstruction. Attached Figure Description

[0045] Figure 1 This is a flowchart of a three-dimensional reconstruction method based on monocular video provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram of the structure of a 3D reconstruction system based on monocular video provided in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0049] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0050] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0052] In related technologies, 3D reconstruction methods typically rely on acquiring large amounts of 3D point cloud or image data using multiple stereo cameras and other equipment, then converting this data into a 3D model using 3D reconstruction algorithms. However, in practical applications, it has been found that these methods suffer from several drawbacks in small spatial environments. The narrow field of view leads to increased feature repetition, causing registration ambiguity. Furthermore, the limited range of motion reduces the effectiveness of motion parallax, exacerbating monocular scale drift and impacting the efficiency of 3D reconstruction. In summary, the technical problems in these related technologies need to be addressed. For example, while some technologies employ Simultaneous Localization and Mapping (SLAM) for 3D reconstruction, this method requires offline processing and cannot meet real-time requirements. Real-time dense SLAM systems also suffer from deficiencies in reconstruction accuracy and completeness. Depth sensor-based solutions are costly and environmentally limited.

[0053] In view of this, this application provides a method, system, device, and medium for 3D reconstruction based on monocular video. This scheme obtains monocular video by acquiring and processing data from an indoor space using a monocular camera. Video segments are obtained by sliding window segmentation of the monocular video based on the spatial volume of the indoor space. The window length can be adaptively adjusted according to the spatial volume, increasing the local window overlap rate and reducing monocular scale drift, thus providing a data foundation for subsequent local reconstruction. Furthermore, this scheme obtains a locally reconstructed point cloud by performing local reconstruction processing on the video segments, obtains a registered scene frame by performing keyframe joint registration processing on the locally reconstructed point cloud, and obtains the 3D reconstruction result by performing global scene optimization processing on the registered scene frame based on spatial constraints. This allows for detection of reconstruction completeness based on spatial constraints, reducing registration errors and improving the accuracy of 3D reconstruction.

[0054] This application provides a 3D reconstruction method based on monocular video, relating to the field of computer vision technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a 3D reconstruction method based on monocular video, but is not limited to the above forms.

[0055] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0056] Figure 1 This is an optional flowchart of a 3D reconstruction method based on monocular video provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0057] Step S101: Data acquisition and processing of the indoor space is performed using a monocular camera to obtain monocular video;

[0058] Step S102: Perform sliding window segmentation on the monocular video according to the spatial volume of the indoor space to obtain video segments;

[0059] Step S103: Perform local reconstruction processing on the video segment to obtain a local reconstructed point cloud;

[0060] Step S104: Perform keyframe joint registration processing on the local reconstructed point cloud to obtain the registered scene frame;

[0061] Step S105: Perform global scene optimization processing on the registered scene frame according to spatial constraints to obtain the three-dimensional reconstruction result.

[0062] Steps S101 to S105 of this embodiment, in a small space environment, use only a common monocular camera to perform high-precision, high-completeness real-time 3D reconstruction. In this embodiment, "small space" specifically refers to an enclosed indoor space with an area of ​​10-50 square meters and a height not exceeding 5 meters. Specifically, this embodiment uses a monocular camera to collect and process data from the indoor space to obtain monocular video. Then, the length of the sliding window is adaptively adjusted according to the space volume, and the monocular video is segmented through the sliding window, dividing the input video stream into overlapping short segments. Local reconstruction processing is then performed on the video segments, and a locally reconstructed point cloud is obtained through an improved multi-branch neural network model. This model can directly predict the dense 3D point cloud map of each frame within the window, and uses intermediate frames as keyframes to establish a local coordinate system. This embodiment also performs keyframe joint registration processing on the locally reconstructed point cloud, incrementally registering the locally reconstructed point cloud to the global coordinate system. Based on visual similarity and baseline suitability scores, historical scene frames related to the current frame can be retrieved from the buffer pool, thereby jointly registering the scene frame. Finally, global scene optimization processing is performed on the registered scene frames based on spatial constraints to obtain the 3D reconstruction result. This embodiment of the application specifically optimizes the following parameters for small spatial environments: the window length is set to 11 frames to balance reconstruction quality and efficiency; the scene frame buffer pool size is set to 30-50 frames; 5-10 of the most relevant scene frames are retrieved for each registration, and a multi-keyframe co-registration strategy is used for registration processing.

[0063] One of the above technical solutions has the following advantages or beneficial effects: The embodiments of this application can reduce registration ambiguity and monocular scale drift through dynamic window segmentation strategy and spatial topology constraints, thereby improving the accuracy of 3D reconstruction.

[0064] In some embodiments, the step of performing sliding window segmentation on the monocular video based on the spatial volume of the indoor space to obtain video segments includes the following steps:

[0065] The volume of the indoor space is calculated based on the monocular video.

[0066] The sliding window is initialized based on the spatial volume, and the length of the sliding window is adjusted based on motion blur detection to obtain the target sliding window;

[0067] The monocular video is segmented according to the target sliding window to obtain the video segment.

[0068] In this embodiment, the volume of an indoor space can be calculated using acquired monocular video, or the volume can be obtained by actual measurement of the indoor space. This embodiment utilizes a deep learning model to perform spatial recognition processing on the monocular video, detecting walls or floors within the space and predicting the depth of the space using a depth prediction model. Based on the detected area and depth, the volume is calculated to obtain the spatial volume. Then, a sliding window is initialized based on the spatial volume, for example, the number of window frames. V represents the spatial volume. Then, based on motion blur detection, the length of the sliding window is adjusted. The degree of blur can be detected by using the image's sharpness or edge information, and the window length is adaptively adjusted according to the degree of blur. The monocular video is then segmented to obtain video segments based on the adaptively adjusted target sliding window.

[0069] One of the above technical solutions has the following advantages or beneficial effects: The embodiments of this application can dynamically adjust the sliding window to adaptively adjust the window size according to the scene requirements, thereby achieving the optimal balance between resources and performance and providing a data foundation for subsequent 3D reconstruction.

[0070] In some embodiments, the process of performing local reconstruction processing on the video segment to obtain a locally reconstructed point cloud includes the following steps:

[0071] The video segment is processed by an image encoder to extract features, and the extracted features are fused by a gated loop unit to obtain spatial scene features.

[0072] The spatial scene features are fused using a keyframe decoder to obtain multi-view information.

[0073] By supporting frame decoder to supplement keyframe information of the spatial scene features, keyframe information is obtained;

[0074] Based on spatial locality constraints, bidirectional cross-attention calculation is performed on the multi-view information and the keyframe information to obtain fused features;

[0075] The point cloud regression module based on deformable convolution performs regression prediction processing on the fused features to obtain the local reconstructed point cloud.

[0076] In this embodiment, the video segment is processed by an image encoder for feature extraction. To address the high repetition of features in small-space scenes, a cross-window feature transfer mechanism is introduced. A gated loop unit is used to perform temporal feature fusion processing on the extracted features. The formula for the gated loop unit is as follows:

[0077] z t =σ(W z ·[h t-1 ,x t ])

[0078] In the formula, z t W represents the update gate output vector at the current time step, σ represents the sigmoid activation function, and W represents the output vector of the current time step. z h represents the learnable weight matrix corresponding to the update gate. t-1 Let x represent the hidden state vector of the previous time step. t This represents the input feature vector at the current time step. Then, the spatial scene features are fused using a keyframe decoder to obtain multi-view information, and keyframe information is supplemented using a support frame decoder to obtain keyframe information. This embodiment also designs a spatial locality constraint for the bidirectional cross-attention mechanism, limiting the attention radius based on the characteristics of small spatial environments. The formula for calculating the attention radius is as follows:

[0079]

[0080] Where r is the attention radius, and W and H are the image width and height, respectively, this embodiment uses the attention radius to force attention to local geometric relationships in small-space scenes. It should be noted that this embodiment uses intermediate frames as keyframes to establish a local coordinate system. Finally, a point cloud regression module based on deformable convolution performs regression prediction processing on the fused features to obtain the locally reconstructed point cloud. Specifically, when adding a thinning module based on deformable convolution to the point cloud regression module, this embodiment limits the kernel deformation offset according to the characteristics of small spaces. The formula for calculating the kernel deformation offset is as follows:

[0081]

[0082] Where Δp is the kernel deformation offset, D is the scene depth estimate, and f is the focal length parameter. In this embodiment, the kernel deformation offset is restricted to ensure that the local optimization in a small space conforms to the spatial scale constraint.

[0083] One of the above technical solutions has the following advantages or beneficial effects: By performing local reconstruction of video segments and adding corresponding spatial scale constraints in combination with the characteristics of small space environments, the embodiments of this application can better detect features in narrow fields of view and improve the accuracy of feature extraction and reconstruction.

[0084] In some embodiments, the keyframe joint registration process of the locally reconstructed point cloud to obtain a registered scene frame includes the following steps:

[0085] Obtain the scene frame buffer pool, which includes historical scene frames;

[0086] The local reconstructed point cloud is subjected to coordinate transformation to obtain global point cloud data;

[0087] The global point cloud data is registered and retrieved based on the scene frame buffer pool to obtain the registered scene frame.

[0088] In this embodiment, the scene frame buffer pool is a pre-built buffer pool containing multiple historical scene frames. These historical scene frames can be obtained from a database or by pre-processing spatial data acquisition. This embodiment incrementally registers the locally reconstructed point cloud to the global coordinate system to obtain global point cloud data. Then, it performs registration retrieval on the global point cloud data based on the scene frame buffer pool. Cross-window feature reuse is achieved through the pre-built scene frame buffer library. Cosine similarity is used to retrieve multiple similar historical scene frames as keyframes for registration, thereby obtaining the registered scene frame.

[0089] One of the above technical solutions has the following advantages or beneficial effects: By performing registration and retrieval processing on global point cloud data, this application embodiment can perform batch multi-keyframe joint registration, thereby improving the efficiency of registration.

[0090] In some embodiments, the step of performing registration retrieval processing on the scene frame buffer pool based on the global point cloud data to obtain the registered scene frame includes the following steps:

[0091] Based on the global point cloud data, cosine similarity retrieval is performed on each historical scene frame in the scene frame buffer pool to generate a keyframe set.

[0092] Spatiotemporal feature alignment processing is performed on the keyframe set to obtain cross-keyframe spatiotemporal features;

[0093] Based on the cross-keyframe spatiotemporal features, the keyframe set is subjected to 3D point cloud registration processing to obtain a registered point cloud.

[0094] The registered point cloud is subjected to point cloud fusion processing to obtain the registered scene frame.

[0095] In this embodiment, the similarity of each historical scene frame in the scene frame buffer pool is calculated based on cosine similarity of the global point cloud data. A similarity threshold can be set for filtering to generate a keyframe set. Then, spatiotemporal feature alignment is performed on the keyframe set. This can be achieved by constructing a spatiotemporal feature cube and extracting features from the cube using a 3D convolutional kernel to obtain cross-keyframe spatiotemporal features. Finally, 3D point cloud registration is performed on the keyframe set based on the cross-keyframe spatiotemporal features. An improved 3D point cloud registration (ICP) algorithm is used for joint registration of multiple keyframes, with the objective function being:

[0096]

[0097] Where the weight w k The confidence level of the keyframes is dynamically calculated, where R represents the rotation transformation matrix. Let t represent the coordinates of the i-th 3D point in the k-th keyframe of the source point cloud, and let t represent the translation transformation vector. Indicates the target point cloud and The corresponding 3D point coordinates. Finally, the registered point cloud is fused to obtain the registered scene frame. Specifically, point cloud fusion can be performed by establishing a probabilistic fusion model. The expression of the fusion model is shown below:

[0098]

[0099] In the formula, p(x) represents the fusion probability density function of a point x in three-dimensional space, and α kThis represents the mixing weight coefficient of the k-th Gaussian component. μ represents the probability density function of a three-dimensional Gaussian distribution. k Let Σ represent the mean vector of the k-th Gaussian component. k This represents the covariance matrix of the k-th Gaussian component. In this embodiment, the optimal fusion parameters can also be solved using the expectation-maximization algorithm, and a point cloud confidence threshold of 3 can be set to filter out low-quality reconstructed point cloud data.

[0100] One of the above technical solutions has the following advantages or beneficial effects: By adopting a multi-keyframe co-registration strategy, the embodiments of this application can simultaneously perform registration processing on multiple keyframes, thereby improving the efficiency and accuracy of registration.

[0101] In some embodiments, the step of performing global scene optimization processing on the registered scene frame according to spatial constraints to obtain the 3D reconstruction result includes the following steps:

[0102] The registered scene frame is subjected to point cloud optimization processing to obtain point cloud optimization data;

[0103] The point cloud optimization data is subjected to planar constraint optimization processing to obtain planar optimization data;

[0104] The planar optimization data is subjected to spatial topology optimization processing to obtain the three-dimensional reconstruction result.

[0105] In this embodiment, a small-space optimization strategy is also set according to the characteristics of small-space scenes. Point cloud distribution optimization is introduced to perform point cloud optimization processing on the registered scene frames to obtain optimized point cloud data. Planar constraint optimization is then performed on the optimized point cloud data to obtain planar optimized data. For example, large planes such as walls are added for constraint optimization. Spatial topology optimization can also be introduced to perform spatial topology optimization processing on the planar optimized data to obtain the 3D reconstruction result. Specifically, point cloud distribution optimization uses a density-aware clustering algorithm, optimizing the point cloud data by defining a density metric. The expression for the density metric is as follows:

[0106]

[0107] In the formula, ρ(x) represents the density measure, x represents the coordinates of the target three-dimensional point whose density is to be calculated, and x i This represents the coordinates of the i-th neighboring point within the neighborhood N(x). Then, an adaptive neighborhood radius r = μ is set. d +ασ d , where μ d Let α represent the average nearest neighbor distance, and σ represent... dThe distribution of point cloud data is optimized based on density metrics and adaptive neighborhood radii. This application embodiment also uses a multi-plane detection algorithm for plane constraint optimization, wherein the expression of the plane detection algorithm is as follows:

[0108]

[0109] In the formula, n represents the plane normal vector, and d represents the distance from the plane to the origin. The gradient term of the normal vector in space is represented by λ, and the regularization coefficient is represented by λ. In this embodiment, a planar relationship graph is established, and orthogonal constraints are enforced, such as an orthogonal constraint of 90±5° between walls, thereby obtaining optimized planar data by applying planar constraints to the point cloud data.

[0110] One of the above technical solutions has the following advantages or beneficial effects: The embodiments of this application can improve the accuracy of three-dimensional reconstruction by introducing planar orthogonality constraint optimization and density adaptive topology optimization.

[0111] In some embodiments, performing spatial topology optimization processing on the planar optimization data to obtain the three-dimensional reconstruction result includes the following steps:

[0112] The planar optimization data is subjected to topology construction processing to obtain a scene topology map;

[0113] The scene topology graph is iteratively optimized using a graph convolutional network to obtain the 3D reconstruction result.

[0114] In this embodiment, a scene topology graph G = (V, E, W) is constructed from the planar optimization data, where vertices represent spatial units. Then, the point cloud distribution is optimized using a graph convolutional network (GCN), and the optimization formula is shown below:

[0115]

[0116] In the formula, H (l+1) This represents the node feature matrix of the (l+1)th layer, and σ represents the nonlinear activation function. The embodiment in this application uses the ReLU activation function. The normalized form of the degree matrix is ​​represented. H represents the normalized form of the adjacency matrix. (l) W represents the input node features of the l-th layer. (l) Let represent the trainable weight matrix of the l-th layer.

[0117] One of the above technical solutions has the following advantages or beneficial effects: The embodiments of this application optimize the spatial topology of point cloud data through graph convolutional networks, which makes the reconstructed data more consistent with the small space environment and improves the accuracy of three-dimensional reconstruction.

[0118] The solutions of this application embodiment will be described in detail and explained below with reference to specific application examples:

[0119] This embodiment of the application receives monocular RGB video input, initializes a first window, attempts all frames as keyframe candidates, selects the reconstruction result with the highest total confidence to initialize the global scene, and optimizes by establishing small-space prior constraints to obtain the 3D reconstruction result. This embodiment of the application acquires video segments through a sliding window, extracts features from each frame through an image encoder, fuses multi-view information through a keyframe decoder, supplements keyframe information through a support frame decoder, and predicts 3D point clouds and confidence through a regression head. Then, the reconstructed point cloud data is registered to the global coordinate system, relevant historical scene frames are retrieved through a scene frame buffer pool, the coordinate system is transformed by a jointly encoded image and geometric feature registration decoder, the global scene is optimized through a scene decoder, and the scene frame buffer pool is updated. Then, optimization is performed through a small-space optimization strategy, introducing planar orthogonality constraints to eliminate wall registration errors. Specifically, this embodiment of the application adopts a two-level neural network framework, segments the video into short segments through a sliding window mechanism, directly predicts local 3D point clouds using the first-level network, and then incrementally registers them to the global coordinate system through the second-level network. Specifically optimized for small-space environments, window size, scene frame management strategies, and spatial constraints enable high-quality real-time reconstruction without explicit camera parameter estimation. By introducing spatially volume-aware window segmentation, planar orthogonality constraint optimization, and density-adaptive topology optimization, compared to related 3D reconstruction methods, the reconstruction completeness is improved by 42.7% in a standard 5m×5m test scene, the registration error is reduced to 0.11m, and the real-time performance of 23FPS is maintained.

[0120] Please see Figure 2 This application also provides a 3D reconstruction system based on monocular video, which can implement the above-mentioned 3D reconstruction method based on monocular video. The system includes:

[0121] The first module 201 is used to acquire and process video of the indoor space through a monocular camera to obtain monocular video.

[0122] The second module 202 is used to perform sliding window segmentation on the monocular video according to the spatial volume of the indoor space to obtain video segments;

[0123] The third module 203 is used to perform local reconstruction processing on the video segment to obtain a local reconstructed point cloud;

[0124] The fourth module 204 is used to perform keyframe joint registration processing on the local reconstructed point cloud to obtain a registered scene frame;

[0125] The fifth module 205 is used to perform global scene optimization processing on the registered scene frame according to spatial constraints to obtain the three-dimensional reconstruction result.

[0126] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0127] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described 3D reconstruction method based on monocular video. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0128] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0129] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0130] The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0131] The memory 302 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and called and executed by the processor 301 to implement the monocular video-based 3D reconstruction method of the embodiments of this application.

[0132] Input / output interface 303 is used to implement information input and output;

[0133] The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0134] Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304);

[0135] The processor 301, memory 302, input / output interface 303, and communication interface 304 are connected to each other within the device via bus 305.

[0136] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described three-dimensional reconstruction method based on monocular video.

[0137] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0138] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0139] This application provides a method, system, device, and medium for 3D reconstruction based on monocular video. The scheme acquires and processes data from an indoor space using a monocular camera to obtain monocular video. Video segments are obtained by sliding window segmentation based on the spatial volume of the indoor space. The window length can be adaptively adjusted according to the spatial volume, increasing the local window overlap rate and reducing monocular scale drift, thus providing a data foundation for subsequent local reconstruction. Furthermore, the scheme obtains a local reconstructed point cloud by performing local reconstruction processing on the video segments. Keyframe joint registration processing is then performed on the local reconstructed point cloud to obtain a registered scene frame. Global scene optimization processing is then performed on the registered scene frame based on spatial constraints to obtain the 3D reconstruction result. This allows for detection of reconstruction completeness based on spatial constraints, reducing registration errors and improving the accuracy of 3D reconstruction.

[0140] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0141] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0142] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0144] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0145] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0146] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0147] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0149] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A 3D reconstruction method based on monocular video, characterized in that, The method includes the following steps: The indoor space is collected and processed using a monocular camera to obtain monocular video. The monocular video is segmented using a sliding window based on the spatial volume of the indoor space to obtain video segments. The video segment is subjected to local reconstruction processing to obtain a locally reconstructed point cloud; The locally reconstructed point cloud is subjected to keyframe joint registration processing to obtain the registered scene frame; Global scene optimization processing is performed on the registered scene frames based on spatial constraints to obtain the 3D reconstruction results; The process of performing sliding window segmentation on the monocular video based on the spatial volume of the indoor space to obtain video segments includes the following steps: The volume of the indoor space is calculated based on the monocular video. The sliding window is initialized based on the spatial volume, and the length of the sliding window is adjusted based on motion blur detection to obtain the target sliding window; The monocular video is segmented according to the target sliding window to obtain the video segment.

2. The method according to claim 1, characterized in that, The process of performing local reconstruction processing on the video segment to obtain a locally reconstructed point cloud includes the following steps: The video segment is processed by an image encoder to extract features, and the extracted features are fused by a gated loop unit to obtain spatial scene features. The spatial scene features are fused using a keyframe decoder to obtain multi-view information. By supporting frame decoder to supplement keyframe information of the spatial scene features, keyframe information is obtained; Based on spatial locality constraints, bidirectional cross-attention calculation is performed on the multi-view information and the keyframe information to obtain fused features; The point cloud regression module based on deformable convolution performs regression prediction processing on the fused features to obtain the local reconstructed point cloud.

3. The method according to claim 1, characterized in that, The process of performing keyframe joint registration on the locally reconstructed point cloud to obtain registered scene frames includes the following steps: Obtain the scene frame buffer pool, which includes historical scene frames; The local reconstructed point cloud is subjected to coordinate transformation to obtain global point cloud data; The global point cloud data is registered and retrieved based on the scene frame buffer pool to obtain the registered scene frame.

4. The method according to claim 3, characterized in that, The process of registering and retrieving the scene frame buffer based on the global point cloud data to obtain the registered scene frame includes the following steps: Based on the global point cloud data, cosine similarity retrieval is performed on each historical scene frame in the scene frame buffer pool to generate a keyframe set. Spatiotemporal feature alignment processing is performed on the keyframe set to obtain cross-keyframe spatiotemporal features; Based on the cross-keyframe spatiotemporal features, the keyframe set is subjected to 3D point cloud registration processing to obtain a registered point cloud. The registered point cloud is subjected to point cloud fusion processing to obtain the registered scene frame.

5. The method according to claim 1, characterized in that, The process of performing global scene optimization on the registered scene frame based on spatial constraints to obtain the 3D reconstruction result includes the following steps: The registered scene frame is subjected to point cloud optimization processing to obtain point cloud optimization data; The point cloud optimization data is subjected to planar constraint optimization processing to obtain planar optimization data; The planar optimization data is subjected to spatial topology optimization processing to obtain the three-dimensional reconstruction result.

6. The method according to claim 5, characterized in that, The process of performing spatial topology optimization on the planar optimization data to obtain the three-dimensional reconstruction result includes the following steps: The planar optimization data is subjected to topology construction processing to obtain a scene topology map; The scene topology graph is iteratively optimized using a graph convolutional network to obtain the 3D reconstruction result.

7. A 3D reconstruction system based on monocular video, characterized in that, The system includes: The first module is used to acquire and process video of the indoor space through a monocular camera to obtain monocular video. The second module is used to perform sliding window segmentation on the monocular video according to the spatial volume of the indoor space to obtain video segments. The third module is used to perform local reconstruction processing on the video segment to obtain a local reconstructed point cloud; The fourth module is used to perform keyframe joint registration processing on the local reconstructed point cloud to obtain the registered scene frame; The fifth module is used to perform global scene optimization processing on the registered scene frame according to spatial constraints to obtain the three-dimensional reconstruction result; The second module is used to perform sliding window segmentation on the monocular video based on the spatial volume of the indoor space to obtain video segments, including: The volume of the indoor space is calculated based on the monocular video. The sliding window is initialized based on the spatial volume, and the length of the sliding window is adjusted based on motion blur detection to obtain the target sliding window; The monocular video is segmented according to the target sliding window to obtain the video segment.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images