Dynamic three-dimensional reconstruction method and system for large scene space

Abstract

The embodiment of the invention provides a large scene space-oriented dynamic three-dimensional reconstruction method and system, and the method comprises the steps: carrying out the continuous shooting of a target construction site through a camera, obtaining a video frame sequence, recognizing a specific target of each video frame according to a target detection model, extracting a key point of the specific target, and carrying out the optimization; constructing a key point pair according to the same specific target key point in the adjacent video frames, constructing a matrix according to the key point pair, decomposing the matrix according to a singular value decomposition algorithm, and determining a preliminary camera pose; re-projection error calculation is carried out on the key point pairs, and the initial camera pose is updated according to the minimum re-projection error to obtain a predicted camera pose; and performing weighted fusion on the predicted camera pose and the real camera pose to obtain a corrected camera pose, performing three-dimensional reconstruction on the construction site image in the video frame sequence according to the corrected camera pose, and determining a corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in a large scene space based on camera pose correction.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a dynamic three-dimensional reconstruction method and system for large scene spaces. Background Art

[0002] 3D reconstruction technology is the process of reconstructing 3D information from single or multiple views. Because single views provide incomplete information, 3D reconstruction is currently more commonly performed based on multiple views. Multi-view 3D reconstruction (similar to binocular positioning) is relatively easy. The method involves first calibrating the camera, calculating the relationship between the camera's image coordinate system and the world coordinate system. Then, the 3D information is reconstructed using information from multiple 2D images.

[0003] In current construction sites, driven by the demands of informatization and digitization, tower cranes are often equipped with surveillance cameras to meet the requirements of 3D reconstruction. Traditional 3D reconstruction methods based on camera motion are categorized into two types: deep learning-based and traditional image processing. Deep learning-based reconstruction systems are currently used in indoor scenarios, but their accuracy is insufficient for large outdoor scenes. Reconstruction methods based on traditional image processing, such as the SFM algorithm (3D reconstruction algorithm), use images from different perspectives and the pixel differences generated by camera capture to reconstruct the 3D space. Camera pose estimation is achieved by reversely matching feature points in the image, using key point detection algorithms such as SIFT and ORB. However, these algorithms often encounter many similar blocks in large scenes, especially construction sites, which can easily lead to mismatches and reconstruction errors. These algorithms cannot meet the requirements for efficient and accurate 3D reconstruction in large scenes. Summary of the Invention

[0004] In response to the problems in the prior art, the present application provides a dynamic three-dimensional reconstruction method and system for large scene spaces, which can improve the efficiency and accuracy of three-dimensional reconstruction in large scene spaces based on camera pose correction.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a dynamic 3D reconstruction method for a large scene space, comprising: Continuously photographing a target construction site using a preset camera to determine a corresponding video frame sequence, performing a target recognition operation on the video frame sequence according to a set target detection model to determine a specific target corresponding to each of the video frames, performing a key point extraction operation on the specific target to determine a corresponding key point, and matching a plurality of key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine a corresponding key point of the specific target, wherein the camera is mounted on a tower boom of a tower crane at the preset construction site to record the rotation of the tower boom; Constructing a key point pair based on the same specific target key point in adjacent video frames, constructing a rotation matrix and translation vector based on the key point pair, and determining the corresponding preliminary camera pose; performing a reprojection error calculation operation on the key point pair to determine a corresponding reprojection error, performing an optimization operation on the reprojection error according to a preset bundle adjustment algorithm, and updating the preliminary camera pose according to a minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; A weighted fusion operation is performed on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, a three-dimensional coordinate point recovery operation is performed on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose to determine the corresponding discrete three-dimensional point cloud data, and a surface reconstruction operation is performed on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine the corresponding three-dimensional model.

[0006] Furthermore, before performing a target recognition operation on the video frame sequence according to the set target detection model to determine the corresponding specific target, the method further includes: Pre-train the preset convolutional neural network model based on the preset open source dataset to determine the corresponding pre-trained model; Collect a large construction site scene data set, perform a specific target labeling operation on the large construction site scene data set, determine a corresponding specific target data set, fine-tune the pre-trained model according to the specific target data set, and determine a corresponding target detection model, wherein the specific target includes at least one of a human body and a vehicle.

[0007] Furthermore, the performing of a key point extraction operation on the specific target to determine the corresponding key points includes: Performing a key point extraction operation on the specific target according to a preset key point prediction network; A post-processing operation is performed on the specific target points after the key point extraction operation according to a preset non-maximum suppression algorithm to determine the corresponding key points.

[0008] Furthermore, performing a reprojection error calculation operation on the key point pair to determine the corresponding reprojection error includes: Determine corresponding projection coordinates according to the two-dimensional coordinates of the key point pairs respectively; Performing a triangulation operation on the key point pairs according to the preliminary camera pose to determine corresponding detection coordinates; An error calculation operation is performed based on the detected coordinates and the projected coordinates to determine a corresponding reprojection error.

[0009] Furthermore, before performing the weighted fusion operation on the predicted camera pose and the set real camera pose, the method further includes: Performing a positioning operation on a preset construction site tower crane according to a preset positioning system to determine the corresponding tower crane position coordinates, performing a reading operation on the tower arm information of the construction site tower crane according to a preset tower arm sensor to determine the corresponding tower arm rotation angle and tower arm posture, and determining the corresponding camera space position coordinates according to the tower crane position coordinates, the tower arm rotation angle, the tower arm posture, and the structural parameters of the construction site tower crane, wherein the camera is installed on the tower arm of the construction site tower crane; The camera posture is read according to a preset camera gimbal control system to determine the corresponding camera posture angle, and the corresponding real camera posture is determined according to the camera space position coordinates and the camera posture angle.

[0010] Furthermore, performing a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose to determine corresponding discrete three-dimensional point cloud data includes: Performing a matching operation on the construction site image key points in the adjacent video frame sequence to determine corresponding construction site image key point pairs; Performing a three-dimensional space triangulation operation based on the corrected camera pose and the key points of the construction site image to determine corresponding key point triangles, and performing a vertex solving operation on the key point triangles to determine corresponding vertex coordinates; A three-dimensional coordinate point recovery operation is performed according to the two-dimensional coordinates of the key point pairs of the construction site image and the vertex coordinates to determine corresponding discrete three-dimensional point cloud data.

[0011] Furthermore, performing a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine a corresponding three-dimensional model includes: Performing a gridding operation on the discrete three-dimensional point cloud data according to a preset triangulation algorithm to determine a corresponding three-dimensional grid; A mapping operation is performed on the three-dimensional grid according to texture information of the construction site image in the video frame sequence to determine a corresponding three-dimensional model.

[0012] In a second aspect, the present application provides a dynamic three-dimensional reconstruction device for a large scene space, comprising: A construction site image key point extraction module is configured to continuously shoot a target construction site using a preset camera to determine a corresponding video frame sequence, perform a target recognition operation on the video frame sequence according to a set target detection model, determine a specific target corresponding to each video frame, perform a key point extraction operation on the specific target to determine the corresponding key point, and perform a matching operation on multiple key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine the corresponding key point of the specific target, wherein the camera is mounted on the tower arm of a tower crane at the preset construction site to record the rotation of the tower arm; A preliminary camera pose determination module is configured to construct a key point pair based on the same specific target key point in adjacent video frames, construct a rotation matrix and a translation vector based on the key point pair, and determine the corresponding preliminary camera pose; a predicted camera pose determination module, configured to perform a reprojection error calculation operation on the key point pairs to determine a corresponding reprojection error, optimize the reprojection error according to a preset bundle adjustment algorithm, and update the preliminary camera pose according to a minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; The three-dimensional model construction module is used to perform a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, perform a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose, determine the corresponding discrete three-dimensional point cloud data, perform a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm, and determine the corresponding three-dimensional model.

[0013] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the dynamic three-dimensional reconstruction method for a large scene space are implemented.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the dynamic three-dimensional reconstruction method for a large scene space.

[0015] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the dynamic three-dimensional reconstruction method for a large scene space.

[0016] It can be seen from the above technical solution that the present application provides a dynamic 3D reconstruction method and system for large scene spaces, which continuously shoots the target construction site with a camera to obtain a video frame sequence, identifies the specific target in each video frame according to the target detection model, extracts the key points of the specific target and optimizes them; constructs key point pairs according to the same specific target key points in adjacent video frames, constructs a matrix according to the key point pairs, decomposes the matrix according to the singular value decomposition algorithm, and determines the preliminary camera pose; calculates the reprojection error of the key point pairs, updates the preliminary camera pose according to the minimum reprojection error to obtain the predicted camera pose; weightedly fuses the predicted camera pose and the real camera pose to obtain the corrected camera pose, and performs 3D reconstruction of the construction site image in the video frame sequence according to the corrected camera pose to determine the corresponding 3D model, thereby improving the efficiency and accuracy of 3D reconstruction in large scene spaces based on camera pose correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is one of the flow charts of the dynamic 3D reconstruction method for a large scene space in an embodiment of the present application; Figure 2 This is the second flow chart of the dynamic 3D reconstruction method for a large scene space in an embodiment of the present application; Figure 3 This is the third flow chart of the dynamic 3D reconstruction method for a large scene space in an embodiment of the present application; Figure 4 This is a fourth flow chart of a dynamic 3D reconstruction method for a large scene space in an embodiment of the present application; Figure 5 This is the fifth flow chart of the dynamic 3D reconstruction method for a large scene space in the embodiment of the present application; Figure 6 This is the sixth flow chart of the dynamic 3D reconstruction method for a large scene space in the embodiment of the present application; Figure 7 This is the seventh flow chart of the dynamic 3D reconstruction method for a large scene space in the embodiment of the present application; Figure 8 This is a structural diagram of a dynamic three-dimensional reconstruction device for a large scene space in an embodiment of the present application; Figure 9 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0019] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0020] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0022] Considering that there are many similar blocks in the large scene environment of the construction site, including similar tower cranes and buildings, the traditional method of three-dimensional reconstruction based on the difference in picture pixels to determine the feature points is prone to matching errors and cannot meet the requirements of efficient and accurate three-dimensional reconstruction in large scene spaces. The present application provides a dynamic three-dimensional reconstruction method and system for large scene spaces, which continuously shoots the target construction site with a camera to obtain a video frame sequence, identifies the specific target of each video frame according to the target detection model, extracts the key points of the specific target and optimizes them; constructs key point pairs according to the same specific target key points in adjacent video frames, constructs a matrix according to the key point pairs, decomposes the matrix according to the singular value decomposition algorithm, and determines the preliminary camera pose; calculates the reprojection error of the key point pairs, updates the preliminary camera pose according to the minimum reprojection error to obtain a predicted camera pose; weightedly fuses the predicted camera pose and the real camera pose to obtain a corrected camera pose, and performs three-dimensional reconstruction of the construction site image in the video frame sequence according to the corrected camera pose to determine the corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in large scene spaces based on camera pose correction.

[0023] In order to improve the efficiency and accuracy of 3D reconstruction in large scene spaces based on camera pose correction, this application provides an embodiment of a dynamic 3D reconstruction method for large scene spaces, see Figure 1 The dynamic 3D reconstruction method for large scene space specifically includes the following contents: Step S101: Continuously photographing a target construction site using a preset camera to determine a corresponding video frame sequence, performing a target recognition operation on the video frame sequence according to a set target detection model to determine a specific target corresponding to each of the video frames, performing a key point extraction operation on the specific target to determine a corresponding key point, and matching a plurality of key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine a corresponding key point of the specific target, wherein the camera is mounted on a tower boom of a tower crane at the preset construction site to record the rotation of the tower boom; Optionally, in steps S101-S104 of this embodiment, the movement of the tower arm camera along with the tower arm is combined, and the camera pose is obtained based on the key point information of specific targets between adjacent video frames, thereby realizing three-dimensional reconstruction of the sweeping area of the tower crane boom, and providing data and navigation basis for subsequent path planning and automatic obstacle avoidance of the hook.

[0024] In step S101, specific targets are detected in successive frames of the construction site scene using a pre-trained target detection model. Significant feature points are extracted, laying the foundation for subsequent camera pose estimation based on the relationships between these feature points. As can be seen, since the position of the target in the image remains relatively constant across consecutive video frames, the camera pose can be estimated based on the offset of these key points, avoiding the mismatching caused by similar patches in traditional methods.

[0025] Optionally, in this embodiment, one way of performing target detection and key point extraction is to perform target detection and key point extraction separately.

[0026] Specifically, first, based on the target detection model to identify specific targets, select an appropriate deep learning framework and algorithm to build the target detection model. Preferably, pre-train the convolutional neural network (CNN) model using large datasets (such as COCO and ImageNet) to improve the model's generalization capabilities. Then, fine-tune the pre-trained model using labeled target scene data to better adapt it to the target detection task.

[0027] Optional, labeled target scene data. Specifically, collect large-scale construction site image data and label a large number of targets including human bodies and vehicles. Train the deep learning model so that it can accurately detect specific targets in the picture, laying the foundation for subsequent key point detection and extraction of the identified specific targets.

[0028] Specifically, secondly, key point extraction is performed. Key points usually refer to points on the target that have significant features and relatively stable positions, such as the top of the head and soles of the feet of a person, the front and rear of a vehicle, etc.

[0029] The first approach is to extract key points of a specific target based on a predefined key point model (such as a human skeleton model). The model will output the two-dimensional coordinates of the key points and their relative positions in the target.

[0030] The second method is to use algorithms such as SIFT and ORB to extract feature points from specific targets in the image to facilitate subsequent matching and reconstruction.

[0031] A third, preferred approach is to use a deep learning model (such as a pose estimation model) to extract key points of specific targets. This model includes a feature extraction network, a key point prediction network, and a post-processing module, which automatically learns and predicts the positional features of key points.

[0032] Specifically, first, the training set of the model collects a large amount of image data containing target objects (such as human bodies, vehicles, etc.), and then annotates the image data to mark the key point positions of each target object (such as human body joints, vehicle feature points, etc.); performs preprocessing operations such as normalization and denoising on the image data to improve the model training effect; constructs a feature extraction network to extract high-level feature representations from the input image; based on the feature extraction network, constructs a key point prediction network, which receives the feature map output by the feature extraction network and outputs the predicted position of each key point; uses the non-maximum suppression (NMS) algorithm to post-process the output of the key point prediction network to further improve the accuracy and robustness of the key point prediction.

[0033] Alternatively, in this embodiment, another approach to target detection and key point extraction is to simultaneously perform specific target detection and key point extraction. This means that the training model not only detects targets but also extracts their key points. Specifically, for the training set, the model's training set annotation information should identify each target object and its key point locations (e.g., human joints, vehicle feature points, etc.) to enable synchronous output of the trained model.

[0034] Understandably, in large outdoor scenes, especially complex environments like construction sites, traditional methods like SIFT and ORB are prone to mismatching, leading to reconstruction errors. Therefore, combining deep learning with object recognition and key point extraction can improve the accuracy of camera pose estimation.

[0035] Optionally, in this embodiment, after the key points of a specific target are extracted, the extracted key points may contain noise or errors, and therefore need to be optimized and screened.

[0036] Specifically, key point optimization uses a nearest neighbor matching algorithm to establish key point correspondences for the same target in consecutive frames, thereby eliminating noise points. The Euclidean distance between the feature point in the current frame and the feature point of the tracked target in the previous frame is calculated, and the point with the smallest distance is considered the matching point.

[0037] For example, in continuous video frames, the key points of a human body are tracked, the Euclidean distance between the key points of the human body in the current frame and the key points of the human body determined in the previous frame is calculated, the closest point pair is found, and the corresponding relationship is established.

[0038] In the presence of noise points, their distance characteristics often differ significantly from those of true target points. While the distances between true target keypoints are within a reasonable range, the distances between noise points and target keypoints may be too large or too small, inconsistent with the normal distance patterns between target keypoints. A reasonable distance threshold is set. During the nearest neighbor matching process, when calculating the distance between a point in the current frame and a target keypoint identified in the previous frame, if this distance is greater than the set threshold, the point is considered a noise point and excluded.

[0039] Specifically, key point screening is performed based on factors such as key point confidence and position stability to retain high-quality key points.

[0040] During model-based keypoint extraction, the model outputs information such as keypoint coordinates and confidence levels. Confidence levels are a measure of the reliability of keypoints. Generally, a higher confidence level indicates a higher likelihood that the keypoint is a valid feature point. Setting a confidence threshold and retaining only keypoints with a confidence level above the threshold can help reduce the number of misidentified keypoints and improve the accuracy of subsequent matching based on the optimized keypoints.

[0041] Position stability refers to the relative stability of a keypoint's position under various conditions, such as image transformation and noise. For high-quality keypoints, their positions should not change dramatically due to minor interference. Because the movement of specific objects between adjacent video frames is minimal, keypoints with drastic position changes between frames can be filtered out.

[0042] After these steps, the optimized and filtered target key points are output as a set of 2D coordinates for subsequent camera pose estimation and 3D reconstruction. These key points serve as important input data for the inverse image processing method, which is crucial for improving the accuracy and robustness of 3D reconstruction.

[0043] Step S102: constructing a key point pair based on the same specific target key point in adjacent video frames, constructing a rotation matrix and a translation vector based on the key point pair, and determining the corresponding preliminary camera pose; Optionally, in step S101 , key point information is obtained, that is, the two-dimensional coordinate set of the key points of the t-th frame image can be expressed as {(x1, y1), (x2, y2), …, (xn, yn)}.

[0044] Optionally, in this embodiment, the extracted key points are used to estimate the camera pose. First, the same key points of adjacent frames are matched. Different from the purpose of matching key points of adjacent frames in step S101, which is to optimize and filter key points and retain the best key points, the purpose of matching key points of adjacent frames in this step is to obtain the best key point pair after matching the best key points, construct the corresponding basic matrix, and obtain preliminary camera pose parameters through matrix operation and decomposition, so as to lay the foundation for subsequent optimization of camera pose parameters to obtain camera pose and establish a three-dimensional model.

[0045] Optionally, in this embodiment, to construct a fundamental matrix, first obtain matching keypoint pairs in adjacent frames. To improve stability and accuracy, the coordinates of the keypoints are translated so that the centroid of the set of matching points in the image is moved to the origin. The coordinate system is then scaled, selecting an appropriate scaling factor so that the average distance from the point to the origin is √2. The processed matching point coordinates are obtained, and a fundamental matrix is constructed. This matrix reflects the epipolar geometric relationship between the two views.

[0046] It is understandable that in an image sequence, when an object moves, there is a geometric relationship between adjacent frames. Suppose we have two frames in a video sequence and want to estimate the camera motion or object motion between the two frames. First, we extract matching key point pairs from the two frames and then use the eight-point method to estimate the fundamental matrix. The fundamental matrix can reflect the changes in camera pose and scene structure between the two frames. For example, if the elements in the fundamental matrix change significantly, it may mean that the camera has undergone a significant rotation or translation, or that an object in the scene has undergone significant movement. By analyzing and further processing the fundamental matrix, we can obtain information such as the camera's motion trajectory or the object's motion speed.

[0047] Optionally, in this embodiment, an intrinsic matrix is constructed. This is a special case of the fundamental matrix, defined based on the intrinsic camera parameters (focal length, identical principal point coordinates, and parallel camera optical axes). The essential matrix describes the relationship between the line connecting the optical centers of the two cameras (the baseline) and the plane formed by the points of the same name in three-dimensional space.

[0048] Convert the fundamental matrix to the intrinsic matrix when the cameras have the same intrinsic parameters: E=K T FK1 Among them, E is the essential matrix, F is the basic matrix, and K1 is the camera intrinsic parameter.

[0049] Optionally, in this embodiment, singular value decomposition (SVD) can be performed on the essential matrix E to separate the rotation matrix R and the translation vector t. If multiple (R, t) solutions exist, a selection is made based on prior knowledge of the scene. One approach is to triangulate multiple possible combinations and ensure that the triangulated points are in front of the camera in both camera coordinate systems (i.e., the depth is positive) to determine the correct camera pose.

[0050] For example, based on the data from the camera sensor, some (R, t) solutions that do not conform to the laws of camera motion can be eliminated. In this way, the (R, t) obtained by singular value decomposition can be used to estimate the camera's pose, thereby providing accurate camera position and orientation information for subsequent 3D reconstruction.

[0051] Step S103: performing a reprojection error calculation operation on the key point pair to determine a corresponding reprojection error, performing an optimization operation on the reprojection error according to a preset bundle adjustment algorithm, and updating the preliminary camera pose according to the minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; Optionally, in this embodiment, the purpose of this step is that the preliminary camera pose obtained in step S102 may have errors, and therefore needs to be further optimized by minimizing the reprojection error.

[0052] Optionally, in this embodiment, after initially estimating the camera pose, the camera pose is further precisely adjusted through nonlinear optimization to minimize the reprojection error. The reprojection error refers to the distance between the projection of a 3D point onto the image plane and the actual observed image point.

[0053] Specifically, first, the camera pose (R, t) obtained by preliminary estimation is used as the initial value for optimization.

[0054] Then, define the reprojection error function, for each 3D point Pit (Xit, Yit, Zit) and its corresponding image point π(Pit, Ct), where Ct=(Rt, tt) represents the camera pose. The reprojection error can be expressed as: Ereproj=∑i=1n∥π(Pit, Ct)−(xit, yit)∥2 Among them, (xit, yit) is the detection coordinate of the i-th key point in the t-th frame image.

[0055] The coordinate differences of all matching keypoint pairs are accumulated or summed to obtain the total value of the reprojection error. This total value reflects the accuracy of the current camera pose estimation.

[0056] Finally, the total reprojection error is minimized using the Levenberg-Marquardt algorithm, bundle adjustment (BA) algorithm, or other nonlinear optimization methods. The optimization variables are the camera pose parameters (rotation matrix and translation vector). The algorithm iteratively adjusts R and t until the reprojection error converges to a minimum. Based on the output of the optimization algorithm, the camera's rotation matrix and translation vector are updated to obtain a more accurate camera pose.

[0057] Step S104: performing a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, performing a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose to determine the corresponding discrete three-dimensional point cloud data, performing a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine the corresponding three-dimensional model.

[0058] Optionally, in this embodiment, the purpose of this step is, in addition to the key point information based on the image, to further improve the estimation accuracy of the camera pose by combining sensor data (such as the position coordinates of the tower crane, the rotation angle of the tower arm, etc.), and to perform three-dimensional reconstruction of the captured image through the fused camera pose to obtain a three-dimensional model of the sweeping area of the tower crane's boom, thereby providing data and navigation basis for the path planning and automatic obstacle avoidance of the hook.

[0059] Specifically, setting the real camera pose is to obtain the spatial position coordinates and attitude angle of the camera in real time through sensors.

[0060] Optionally, the camera pose estimated based on the key point information of the image is fused with the camera pose obtained from the sensor data. Methods such as weighted averaging and Kalman filtering can be used to fuse the pose information of the two data sources. Let Ctsensor be the camera pose obtained by the sensor, which can be fused and corrected: Ctfused=αCt+(1−α)Ctsensor Among them, α is the fusion weight, which can be adjusted according to actual conditions.

[0061] Optionally, apply the optimized camera pose to key point reprojection, check whether the reprojection error meets the preset threshold requirement, and check whether the trajectory of the camera pose changing over time is smooth to exclude outliers or jump points.

[0062] Optionally, in this embodiment, 3D reconstruction involves constructing a triangle in 3D space for each pair of matched feature points using their 2D image coordinates and the corresponding fused camera pose. This triangle contains the 3D coordinate information of the feature points. The 3D coordinates of the feature points are obtained by solving the vertex coordinates of the triangle using linear algebra and geometric calculations.

[0063] Optionally, in this embodiment, the point cloud data obtained by triangulation is discrete. In order to obtain a continuous three-dimensional surface model, surface reconstruction is required. First, the point cloud data obtained by triangulation is filtered to remove noise points and outliers, thereby improving the quality of the point cloud. Then, the point cloud data is converted into a three-dimensional mesh model through Delaunay triangulation. Preferably, the three-dimensional mesh model can also be implemented through the Ball-Pivoting algorithm. The meshing process needs to pay attention to maintaining the smoothness and continuity of the mesh. Finally, the texture information in the original image data is mapped to the three-dimensional mesh model to make the reconstructed three-dimensional model more realistic and vivid.

[0064] Optionally, in this embodiment, in order to improve the accuracy and effect of three-dimensional reconstruction, the reconstruction result may be optimized.

[0065] Specifically, global optimization algorithms (such as ICP and bundle adjustment) are used to optimize the overall reconstruction results, improving the accuracy and consistency of the 3D model. Local details in the 3D model are then enhanced, such as by sharpening edges and enhancing textures, to make the model more refined and realistic.

[0066] This example demonstrates how this embodiment combines the movement of the tower boom camera with the tower boom to perform three-dimensional reconstruction of the sweep area of the tower crane boom. Based on the key point information of specific targets between video frames, the camera's position is reversely solved through image analysis, and the camera motion position information obtained by the boom rotation is used for correction. In this way, a three-dimensional model of the sweep area of the tower crane boom is obtained, providing data and navigation basis for subsequent path planning and automatic obstacle avoidance of the hook.

[0067] From the above description, it can be seen that the dynamic three-dimensional reconstruction method for large scene spaces provided in the embodiment of the present application can obtain a video frame sequence by continuously shooting the target construction site with a camera, identify the specific target of each video frame according to the target detection model, extract the key points of the specific target and optimize them; construct key point pairs according to the same specific target key points in adjacent video frames, construct a matrix according to the key point pairs, decompose the matrix according to the singular value decomposition algorithm, and determine the preliminary camera pose; calculate the reprojection error of the key point pairs, update the preliminary camera pose according to the minimum reprojection error to obtain the predicted camera pose; weightedly fuse the predicted camera pose and the real camera pose to obtain the corrected camera pose, perform three-dimensional reconstruction of the construction site image in the video frame sequence according to the corrected camera pose, and determine the corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in large scene spaces based on camera pose correction.

[0068] In one embodiment of the dynamic three-dimensional reconstruction method for a large scene space of the present application, see Figure 2 , and can also include the following: Step S201: performing a pre-training operation on a preset convolutional neural network model according to a preset open source data set to determine a corresponding pre-training model; Step S202: Collect a large construction site scene data set, perform a specific target labeling operation on the large construction site scene data set, determine a corresponding specific target data set, fine-tune the pre-trained model according to the specific target data set, and determine a corresponding target detection model, wherein the specific target includes at least one of a human body and a vehicle.

[0069] Optionally, in this embodiment, a suitable deep learning framework and algorithm are selected to build an object detection model. Preferably, a convolutional neural network (CNN) model is pre-trained using large datasets (such as COCO and ImageNet) to improve the model's generalization capabilities. The pre-trained model is then fine-tuned using labeled object scene data to better adapt the model to the object detection task.

[0070] Optional, labeled target scene data. Specifically, collect large-scale construction site image data and label a large number of targets including human bodies and vehicles. Train the deep learning model so that it can accurately detect specific targets in the picture, laying the foundation for subsequent key point detection and extraction of the identified specific targets.

[0071] Through step S202 , this embodiment obtains a specific target detection model based on the construction site scene, laying a solid foundation for subsequent key point extraction based on the model structure.

[0072] In one embodiment of the dynamic 3D reconstruction method for a large scene space of the present application, see Figure 3, and can also include the following: Step S301: performing a key point extraction operation on the specific target according to a preset key point prediction network; Step S302: performing a post-processing operation on the specific target points after the key point extraction operation according to a preset non-maximum suppression algorithm to determine the corresponding key points.

[0073] Optionally, in this embodiment, this step involves extracting key points from specific objects (such as humans and vehicles) in the video frames using a pre-set key point prediction network. The key point prediction network is a deep learning model trained to automatically identify and locate significant feature points on an object. These feature points are relatively stable and representative locations on the object, such as joints on a human body or characteristic parts of a vehicle.

[0074] The keypoint prediction network receives the target-specific image or video frame output by the target detection model in the previous step and further detects keypoints within the target (such as joints on a human body or feature points on a vehicle). The network first extracts features from the input image, extracting high-level features through structures such as convolutional layers and pooling layers. The network then uses these features to predict the locations of the keypoints using fully connected layers or other regression methods. These locations can be annotated and subsequently converted to two-dimensional coordinates through post-processing, or they can be expressed as two-dimensional coordinates, indicating the location of the keypoints within the image.

[0075] Optionally, for the training of the key point prediction network, the training set collects a large amount of image data containing target objects (such as human bodies, vehicles, etc.), and then annotates the image data to mark the key point positions of each target object (such as human body joints, vehicle feature points, etc.); performs preprocessing operations such as normalization and denoising on the image data to improve the model training effect; constructs a feature extraction network to extract high-level feature representations from the input image; based on the feature extraction network, constructs a key point prediction network, which receives the feature map output by the feature extraction network and outputs the predicted position of each key point; uses the non-maximum suppression (NMS) algorithm to post-process the output of the key point prediction network to further improve the accuracy and robustness of the key point prediction.

[0076] Optionally, after keypoint extraction, multiple candidate keypoints are obtained, which may include some duplicate or incorrect keypoints. To remove these redundant and incorrect keypoints, the Non-Maximum Suppression (NMS) algorithm is used for post-processing.

[0077] First, all candidate keypoints are grouped into a set. A score is calculated for each candidate keypoint, which reflects the probability or confidence that the keypoint is a true keypoint. Then, for each keypoint, a local area (window) around it is checked. If the keypoint has the highest score in the local area, it is retained; otherwise, it is removed from the set. The above process is repeated until all keypoints have been processed.

[0078] Through the non-maximum suppression algorithm, duplicate and erroneous key points can be effectively removed, and the most likely correct key points can be retained, thereby improving the accuracy and robustness of key point extraction.

[0079] Through step S302 , this embodiment implements the key steps of extracting and processing key points of a specific target, thereby providing accurate key point information for subsequent three-dimensional reconstruction.

[0080] In one embodiment of the dynamic 3D reconstruction method for a large scene space of the present application, see Figure 4 , and can also include the following: Step S401: determining corresponding projection coordinates according to the two-dimensional coordinates of the key point pairs; Step S402: performing a triangulation operation on the key point pair according to the preliminary camera pose to determine the corresponding detection coordinates; Step S403: performing an error calculation operation according to the detected coordinates and the projected coordinates to determine a corresponding re-projection error.

[0081] Optionally, in this embodiment, corresponding projection coordinates are determined based on the two-dimensional coordinates of the key point pairs.

[0082] In this step, we need to obtain the 2D coordinates on the image plane for each pair of keypoints (keypoints belonging to the same specific object in adjacent video frames). These coordinates represent the keypoint's location in the image. These 2D coordinates are projected coordinates, meaning they are the result of projecting the 3D world onto the 2D image plane. These are known coordinates directly extracted from the image using the keypoint prediction network.

[0083] Optionally, in this embodiment, a triangulation operation is performed on the key point pairs according to the preliminary camera pose to determine corresponding detection coordinates.

[0084] After obtaining the 2D projection coordinates of the keypoint pairs, the next step is to triangulate the keypoints using the preliminary estimated camera pose (including the rotation matrix and translation vector). Using the preliminary camera pose (rotation matrix and translation vector) obtained in step S102, geometric calculations are performed to back-project the keypoint pairs on the 2D image plane back into 3D space to obtain their corresponding 3D coordinates, which are referred to as the "detection coordinates." This step assumes that the keypoint pairs correspond to the same physical point in 3D space and uses the camera pose parameters and imaging model to determine the 3D position of this point.

[0085] Optionally, in this embodiment, an error calculation operation is performed based on the detection coordinates and the projection coordinates to determine a corresponding re-projection error.

[0086] Reprojection error is an important metric for measuring 3D reconstruction accuracy. In this step, the detected coordinates (3D coordinates) are reprojected onto the 2D image plane using the camera model to obtain theoretical projection coordinates. These theoretical projection coordinates are then compared with the actual projection coordinates (i.e., the coordinates obtained in step S401).

[0087] Specifically, reprojection uses the camera intrinsic parameters and the known camera pose (here the preliminarily estimated camera pose is used) to reproject the 3D detection coordinates back to the 2D image plane to obtain the reprojected coordinates.

[0088] Specifically, error calculation involves calculating the difference between the reprojected coordinates and the actual projected coordinates. This difference is the reprojection error. The reprojection error reflects the accuracy of the 3D reconstruction; the smaller the error, the more accurate the reconstruction result.

[0089] Through step S403, this embodiment successfully calculates the reprojection error under the preliminary camera pose. This error will be used in subsequent optimization steps to minimize the reprojection error by continuously adjusting the camera pose, thereby improving the accuracy of 3D reconstruction.

[0090] In one embodiment of the dynamic 3D reconstruction method for a large scene space of the present application, see Figure 5 , and can also include the following: Step S501: performing a positioning operation on a preset construction site tower crane according to a preset positioning system to determine the corresponding tower crane position coordinates, performing a tower arm information reading operation on the construction site tower crane according to a preset tower arm sensor to determine the corresponding tower arm rotation angle and tower arm posture, and determining the corresponding camera spatial position coordinates according to the tower crane position coordinates, the tower arm rotation angle, the tower arm posture, and the structural parameters of the construction site tower crane, wherein the camera is installed on the tower arm of the construction site tower crane; Step S502: performing a posture reading operation on the camera according to a preset camera gimbal control system to determine a corresponding camera posture angle, and determining a corresponding real camera posture according to the camera spatial position coordinates and the camera posture angle.

[0091] Optionally, in a construction site application scenario, it is easy to obtain the spatial position coordinates of the camera based on various position sensors.

[0092] Specifically, the precise position of the camera in the world coordinate system can be determined based on the position coordinates of the tower crane and the rotation angle of the tower arm. The posture of the camera itself, if it does not have a gimbal, its posture angle is fixed during installation; if it has a gimbal, its posture angle can be obtained through the gimbal, and the precise position of the camera in the world coordinate system can be known.

[0093] Among them, data can be obtained through real-time sensors arranged on the tower crane.

[0094] Through step S502, this embodiment successfully obtains the true camera pose, laying the foundation for subsequent camera pose correction.

[0095] In one embodiment of the dynamic 3D reconstruction method for a large scene space of the present application, see Figure 6 , and can also include the following: Step S601: performing a matching operation on the construction site image key points in the adjacent video frame sequence to determine corresponding construction site image key point pairs; Step S602: performing a three-dimensional space triangulation operation based on the corrected camera pose and the key point pairs of the construction site image to determine corresponding key point triangles, and performing a vertex solving operation on the key point triangles to determine corresponding vertex coordinates; Step S603: performing a three-dimensional coordinate point recovery operation according to the two-dimensional coordinates of the key point pairs of the construction site image and the vertex coordinates to determine corresponding discrete three-dimensional point cloud data.

[0096] Optionally, in this embodiment, a matching operation is performed on the construction site image key points in the adjacent video frame sequence to determine corresponding construction site image key point pairs.

[0097] In this step, key points in the construction site images are matched across a sequence of adjacent video frames. Keypoint detection is performed using a specific target to find corresponding key points between adjacent frames. The positional differences between these key points in adjacent frames reflect subtle camera movements and provide important evidence for subsequent 3D reconstruction.

[0098] Optionally, in this embodiment, a three-dimensional space triangulation operation is performed based on the corrected camera pose and the key points of the construction site image to determine the corresponding key point triangle, and a vertex solving operation is performed on the key point triangle to determine the corresponding vertex coordinates; Specifically, for each pair of matched keypoints, we can assume they correspond to the same point in 3D space. Then, using the 2D coordinates of these two keypoints and the corresponding camera pose, we can construct a 3D triangle. The two bases of this triangle are the projections of the same physical point onto the image plane, as captured by the camera at different positions, while the vertices are the actual positions of this physical point in 3D space.

[0099] Through geometric calculations, we can solve for the coordinates of the triangle's vertices, which is the actual location of the physical point in three-dimensional space. This solves a system of linear equations whose coefficients are determined by the camera's intrinsic and extrinsic parameters (i.e., the camera's pose) and the two-dimensional coordinates of the key points.

[0100] Optionally, in this embodiment, a three-dimensional coordinate point recovery operation is performed based on the two-dimensional coordinates of the key point pairs of the construction site image and the vertex coordinates to determine corresponding discrete three-dimensional point cloud data.

[0101] Specifically, each matched keypoint pair corresponds to a point in 3D space (i.e., a vertex of a triangle). Collecting the 3D coordinates of these points forms a discrete 3D point cloud. This 3D point cloud contains the 3D positional information of all successfully matched keypoints in the scene. It can be used for subsequent 3D modeling, visualization, measurement, and other operations.

[0102] Through step S603, this embodiment successfully recovers three-dimensional spatial information from two-dimensional image information and constructs discrete three-dimensional point cloud data, laying a solid foundation for subsequent three-dimensional reconstruction and scene understanding.

[0103] In one embodiment of the dynamic 3D reconstruction method for a large scene space of the present application, see Figure 7 , and can also include the following: Step S701: performing a gridding operation on the discrete three-dimensional point cloud data according to a preset triangulation algorithm to determine a corresponding three-dimensional grid; Step S702: performing a mapping operation on the three-dimensional grid according to texture information of the construction site image in the video frame sequence to determine a corresponding three-dimensional model.

[0104] Optionally, in this embodiment, a gridding operation is performed on the discrete three-dimensional point cloud data according to a preset triangulation algorithm to determine a corresponding three-dimensional grid.

[0105] Specifically, this step aims to convert the discrete 3D point cloud data obtained in the previous step into a continuous 3D mesh model. While the discrete point cloud contains 3D information about the scene, it is discrete and cannot be directly used for visualization or further analysis. Therefore, a triangulation algorithm is needed to connect these discrete points into a continuous triangular mesh.

[0106] A triangulation algorithm is a method for partitioning a discrete set of points into a set of non-overlapping triangles, ensuring that these triangles cover the entire point cloud area and meet certain optimization criteria (such as maximizing the minimum angle or the Delaunay criterion). Triangulation algorithms include Delaunay triangulation and the Bowyer-Watson algorithm.

[0107] Through triangulation algorithms, discrete 3D point cloud data can be converted into a 3D mesh model composed of triangular facets. This 3D mesh model not only retains the 3D structural information of the original point cloud, but also provides a continuous surface representation, providing the basis for subsequent operations (such as texture mapping and rendering).

[0108] Optionally, in this embodiment, a mapping operation is performed on the three-dimensional grid according to texture information of the construction site image in the video frame sequence to determine a corresponding three-dimensional model.

[0109] Specifically, after obtaining a 3D mesh model, the next step is to enhance the model's realism through texture mapping. Texture mapping is a technique that maps a 2D image (i.e., texture) onto the surface of a 3D model, making the model surface appear richer in detail and more realistic.

[0110] In this step, texture information of the construction site images needs to be extracted from the original video frame sequence. This texture information may include brick texture, wall color, floor material, etc. After the texture information is extracted, it is necessary to map these textures onto the corresponding surfaces of the 3D mesh model using a mapping algorithm.

[0111] Texture mapping methods include UV mapping, normal mapping, and environment mapping. UV mapping is preferred. This method maps textures by associating each vertex of a 3D model with a point (i.e., UV coordinate) on a 2D texture image. During the mapping process, texture transformations such as repetition, scaling, and rotation must be considered to ensure the texture is correctly applied to the model surface.

[0112] Texture mapping can make otherwise monotonous 3D mesh models more vivid and lifelike, more in line with human visual habits. The resulting 3D model contains both the scene's geometric structure information and rich surface details and texture information, making it suitable for a variety of applications such as visualization, simulation, and virtual reality.

[0113] Through step S702, this embodiment successfully constructs a realistic and accurate three-dimensional model of the tower crane boom sweep area, providing data and navigation basis for subsequent path planning and automatic obstacle avoidance of the hook.

[0114] In order to improve the efficiency and accuracy of 3D reconstruction in large scene spaces based on camera pose correction, the present application provides an embodiment of a dynamic 3D reconstruction device for large scene spaces for implementing all or part of the content of the dynamic 3D reconstruction method for large scene spaces, see Figure 8 The dynamic 3D reconstruction device for large scene space specifically includes the following contents: The construction site image key point extraction module 10 is used to continuously shoot the target construction site according to a preset camera, determine the corresponding video frame sequence, perform a target recognition operation on the video frame sequence according to a set target detection model, determine the specific target corresponding to each video frame, perform a key point extraction operation on the specific target to determine the corresponding key point, and perform a matching operation on multiple key points in adjacent video frames according to a preset nearest neighbor matching algorithm to determine the corresponding key point of the specific target, wherein the camera is installed on the tower arm of the preset construction site tower crane to record the rotation of the tower arm; A preliminary camera pose determination module 20 is configured to construct a key point pair based on the same specific target key point in adjacent video frames, construct a rotation matrix and a translation vector based on the key point pair, and determine the corresponding preliminary camera pose; a predicted camera pose determination module 30, configured to perform a reprojection error calculation operation on the key point pairs to determine corresponding reprojection errors, optimize the reprojection errors according to a preset bundle adjustment algorithm, and update the preliminary camera pose according to a minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; The three-dimensional model construction module 40 is used to perform a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, perform a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose, determine the corresponding discrete three-dimensional point cloud data, perform a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm, and determine the corresponding three-dimensional model.

[0115] From the above description, it can be seen that the dynamic three-dimensional reconstruction device for large scene spaces provided by the embodiment of the present application can obtain a video frame sequence by continuously shooting the target construction site with a camera, identify the specific target of each video frame according to the target detection model, extract the key points of the specific target and optimize them; construct key point pairs according to the same specific target key points in adjacent video frames, construct a matrix according to the key point pairs, decompose the matrix according to the singular value decomposition algorithm, and determine the preliminary camera pose; calculate the reprojection error of the key point pairs, update the preliminary camera pose according to the minimum reprojection error to obtain the predicted camera pose; weightedly fuse the predicted camera pose and the real camera pose to obtain the corrected camera pose, perform three-dimensional reconstruction of the construction site image in the video frame sequence according to the corrected camera pose, and determine the corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in large scene spaces based on camera pose correction.

[0116] In order to improve the efficiency and accuracy of 3D reconstruction in large scene spaces based on camera pose correction, the present application provides an embodiment of a dynamic 3D reconstruction system for large scene spaces for implementing all or part of the dynamic 3D reconstruction method for large scene spaces. The dynamic 3D reconstruction system for large scene spaces includes a server and a preset camera communicatively connected to the server, and the server includes: A construction site image key point extraction module is configured to continuously shoot a target construction site using a preset camera to determine a corresponding video frame sequence, perform a target recognition operation on the video frame sequence according to a set target detection model, determine a specific target corresponding to each video frame, perform a key point extraction operation on the specific target to determine the corresponding key point, and perform a matching operation on multiple key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine the corresponding key point of the specific target, wherein the camera is mounted on the tower arm of a tower crane at the preset construction site to record the rotation of the tower arm; A preliminary camera pose determination module is configured to construct a key point pair based on the same specific target key point in adjacent video frames, construct a rotation matrix and a translation vector based on the key point pair, and determine the corresponding preliminary camera pose; a predicted camera pose determination module, configured to perform a reprojection error calculation operation on the key point pairs to determine a corresponding reprojection error, optimize the reprojection error according to a preset bundle adjustment algorithm, and update the preliminary camera pose according to a minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; The three-dimensional model construction module is used to perform a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, perform a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose, determine the corresponding discrete three-dimensional point cloud data, perform a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm, and determine the corresponding three-dimensional model.

[0117] From the above description, it can be seen that the dynamic three-dimensional reconstruction system for large scene spaces provided by the embodiment of the present application can obtain a video frame sequence by continuously shooting the target construction site with a camera, identify the specific target of each video frame according to the target detection model, extract the key points of the specific target and optimize them; construct key point pairs according to the same specific target key points in adjacent video frames, construct a matrix according to the key point pairs, decompose the matrix according to the singular value decomposition algorithm, and determine the preliminary camera pose; calculate the reprojection error of the key point pairs, update the preliminary camera pose according to the minimum reprojection error to obtain the predicted camera pose; weightedly fuse the predicted camera pose and the real camera pose to obtain the corrected camera pose, perform three-dimensional reconstruction of the construction site image in the video frame sequence according to the corrected camera pose, and determine the corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in large scene spaces based on camera pose correction.

[0118] From a hardware perspective, in order to improve the efficiency and accuracy of 3D reconstruction in large scene spaces based on camera pose correction, this application provides an embodiment of an electronic device for implementing all or part of the dynamic 3D reconstruction method for large scene spaces. The electronic device specifically includes the following: A processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to implement information transmission between the dynamic three-dimensional reconstruction method for large scene spaces and related devices such as core business systems, user terminals, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited to this. In this embodiment, the logic controller can be implemented with reference to the embodiment of the dynamic three-dimensional reconstruction method for large scene spaces and the embodiment of the dynamic three-dimensional reconstruction method for large scene spaces in the embodiment, and their contents are incorporated herein, and repeated parts are not repeated.

[0119] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0120] In practical applications, part of the dynamic 3D reconstruction method for large scene spaces can be executed on the electronic device side as described above, or all operations can be completed on the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not impose any restrictions on this. If all operations are completed on the client device, the client device may also include a processor.

[0121] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0122] Figure 9 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 9 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0123] In one embodiment, the dynamic 3D reconstruction method for large scene spaces can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control: Step S101: Continuously photographing a target construction site using a preset camera to determine a corresponding video frame sequence, performing a target recognition operation on the video frame sequence according to a set target detection model to determine a specific target corresponding to each of the video frames, performing a key point extraction operation on the specific target to determine a corresponding key point, and matching a plurality of key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine a corresponding key point of the specific target, wherein the camera is mounted on a tower boom of a tower crane at the preset construction site to record the rotation of the tower boom; Step S102: constructing a key point pair based on the same specific target key point in adjacent video frames, constructing a rotation matrix and a translation vector based on the key point pair, and determining the corresponding preliminary camera pose; Step S103: performing a reprojection error calculation operation on the key point pair to determine a corresponding reprojection error, performing an optimization operation on the reprojection error according to a preset bundle adjustment algorithm, and updating the preliminary camera pose according to the minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; Step S104: performing a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, performing a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose to determine the corresponding discrete three-dimensional point cloud data, performing a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine the corresponding three-dimensional model.

[0124] From the above description, it can be seen that the electronic device provided in the embodiment of the present application continuously shoots the target construction site with a camera to obtain a video frame sequence, identifies the specific target of each video frame according to the target detection model, extracts the key points of the specific target and optimizes them; constructs key point pairs according to the same specific target key points in adjacent video frames, constructs a matrix according to the key point pairs, decomposes the matrix according to the singular value decomposition algorithm, and determines the preliminary camera pose; calculates the reprojection error of the key point pairs, updates the preliminary camera pose according to the minimum reprojection error to obtain the predicted camera pose; weightedly fuses the predicted camera pose and the real camera pose to obtain the corrected camera pose, and performs three-dimensional reconstruction of the construction site image in the video frame sequence according to the corrected camera pose to determine the corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in large scene space based on camera pose correction.

[0125] In another embodiment, the dynamic three-dimensional reconstruction method for large scene space can be configured separately from the central processing unit 9100. For example, the dynamic three-dimensional reconstruction method for large scene space can be configured as a chip connected to the central processing unit 9100, and the function of the dynamic three-dimensional reconstruction method for large scene space can be realized through the control of the central processing unit.

[0126] like Figure 9 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 9 In addition, the electronic device 9600 may also include all components shown in Figure 9 For components not shown, reference may be made to the prior art.

[0127] like Figure 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0128] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0129] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0130] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0131] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0132] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0133] Based on different communication technologies, multiple communication modules 9110 can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is also coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0134] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the method for dynamic 3D reconstruction of a large scene space in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for dynamic 3D reconstruction of a large scene space in the above-mentioned embodiment is implemented. For example, when the processor executes the computer program, the following steps are implemented: Step S101: Continuously photographing a target construction site using a preset camera to determine a corresponding video frame sequence, performing a target recognition operation on the video frame sequence according to a set target detection model to determine a specific target corresponding to each of the video frames, performing a key point extraction operation on the specific target to determine a corresponding key point, and matching a plurality of key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine a corresponding key point of the specific target, wherein the camera is mounted on a tower boom of a tower crane at the preset construction site to record the rotation of the tower boom; Step S102: constructing a key point pair based on the same specific target key point in adjacent video frames, constructing a rotation matrix and a translation vector based on the key point pair, and determining the corresponding preliminary camera pose; Step S103: performing a reprojection error calculation operation on the key point pair to determine a corresponding reprojection error, performing an optimization operation on the reprojection error according to a preset bundle adjustment algorithm, and updating the preliminary camera pose according to the minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; Step S104: performing a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, performing a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose to determine the corresponding discrete three-dimensional point cloud data, performing a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine the corresponding three-dimensional model.

[0135] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application continuously shoots the target construction site with a camera to obtain a video frame sequence, identifies the specific target of each video frame according to the target detection model, extracts the key points of the specific target and optimizes them; constructs key point pairs according to the same specific target key points in adjacent video frames, constructs a matrix according to the key point pairs, decomposes the matrix according to the singular value decomposition algorithm, and determines the preliminary camera pose; calculates the reprojection error of the key point pairs, updates the preliminary camera pose according to the minimum reprojection error to obtain the predicted camera pose; weightedly fuses the predicted camera pose and the real camera pose to obtain the corrected camera pose, and performs three-dimensional reconstruction of the construction site image in the video frame sequence according to the corrected camera pose to determine the corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in large scene space based on camera pose correction.

[0136] Embodiments of the present application also provide a computer program product capable of implementing all steps of the large-scene-space dynamic 3D reconstruction method in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instructions are executed by a processor, the computer program / instructions implement the steps of the large-scene-space dynamic 3D reconstruction method. For example, the computer program / instructions implement the following steps: Step S101: Continuously photographing a target construction site using a preset camera to determine a corresponding video frame sequence, performing a target recognition operation on the video frame sequence according to a set target detection model to determine a specific target corresponding to each of the video frames, performing a key point extraction operation on the specific target to determine a corresponding key point, and matching a plurality of key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine a corresponding key point of the specific target, wherein the camera is mounted on a tower boom of a tower crane at the preset construction site to record the rotation of the tower boom; Step S102: constructing a key point pair based on the same specific target key point in adjacent video frames, constructing a rotation matrix and a translation vector based on the key point pair, and determining the corresponding preliminary camera pose; Step S103: performing a reprojection error calculation operation on the key point pair to determine a corresponding reprojection error, performing an optimization operation on the reprojection error according to a preset bundle adjustment algorithm, and updating the preliminary camera pose according to the minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; Step S104: performing a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, performing a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose to determine the corresponding discrete three-dimensional point cloud data, performing a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine the corresponding three-dimensional model.

[0137] From the above description, it can be seen that the computer program product provided in the embodiment of the present application continuously shoots the target construction site with a camera to obtain a video frame sequence, identifies the specific target of each video frame according to the target detection model, extracts the key points of the specific target and optimizes them; constructs key point pairs according to the same specific target key points in adjacent video frames, constructs a matrix according to the key point pairs, decomposes the matrix according to the singular value decomposition algorithm, and determines the preliminary camera pose; calculates the reprojection error of the key point pairs, updates the preliminary camera pose according to the minimum reprojection error to obtain the predicted camera pose; weightedly fuses the predicted camera pose and the real camera pose to obtain the corrected camera pose, performs three-dimensional reconstruction of the construction site image in the video frame sequence according to the corrected camera pose, and determines the corresponding three-dimensional model, thereby improving the efficiency and accuracy of three-dimensional reconstruction in large scene space based on camera pose correction.

[0138] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0139] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0142] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A dynamic 3D reconstruction method for large scene spaces, characterized in that: The method comprises: Continuously photographing a target construction site using a preset camera to determine a corresponding video frame sequence, performing a target recognition operation on the video frame sequence according to a set target detection model to determine a specific target corresponding to each of the video frames, performing a key point extraction operation on the specific target to determine a corresponding key point, and matching a plurality of key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine a corresponding key point of the specific target, wherein the camera is mounted on a tower boom of a tower crane at the preset construction site to record the rotation of the tower boom; Constructing a key point pair based on the same specific target key point in adjacent video frames, constructing a rotation matrix and translation vector based on the key point pair, and determining the corresponding preliminary camera pose; performing a reprojection error calculation operation on the key point pair to determine a corresponding reprojection error, performing an optimization operation on the reprojection error according to a preset bundle adjustment algorithm, and updating the preliminary camera pose according to a minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; A weighted fusion operation is performed on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, a three-dimensional coordinate point recovery operation is performed on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose to determine the corresponding discrete three-dimensional point cloud data, and a surface reconstruction operation is performed on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine the corresponding three-dimensional model.

2. The method for dynamic 3D reconstruction of a large scene space according to claim 1, characterized in that: Before performing a target recognition operation on the video frame sequence according to the set target detection model to determine the corresponding specific target, the method includes: Pre-train the preset convolutional neural network model based on the preset open source dataset to determine the corresponding pre-trained model; Collect a large construction site scene data set, perform a specific target labeling operation on the large construction site scene data set, determine a corresponding specific target data set, fine-tune the pre-trained model according to the specific target data set, and determine a corresponding target detection model, wherein the specific target includes at least one of a human body and a vehicle.

3. The method for dynamic 3D reconstruction of a large scene space according to claim 1, characterized in that: The performing a key point extraction operation on the specific target to determine the corresponding key points includes: Performing a key point extraction operation on the specific target according to a preset key point prediction network; A post-processing operation is performed on the specific target points after the key point extraction operation according to a preset non-maximum suppression algorithm to determine the corresponding key points.

4. The method for dynamic 3D reconstruction of a large scene space according to claim 1, characterized in that: The performing a reprojection error calculation operation on the key point pair to determine the corresponding reprojection error includes: Determine corresponding projection coordinates according to the two-dimensional coordinates of the key point pairs respectively; Performing a triangulation operation on the key point pairs according to the preliminary camera pose to determine corresponding detection coordinates; An error calculation operation is performed based on the detected coordinates and the projected coordinates to determine a corresponding reprojection error.

5. The method for dynamic 3D reconstruction of a large scene space according to claim 1, characterized in that: Before performing a weighted fusion operation on the predicted camera pose and the set real camera pose, the method includes: Performing a positioning operation on a preset construction site tower crane according to a preset positioning system to determine the corresponding tower crane position coordinates, performing a reading operation on the tower arm information of the construction site tower crane according to a preset tower arm sensor to determine the corresponding tower arm rotation angle and tower arm posture, and determining the corresponding camera space position coordinates according to the tower crane position coordinates, the tower arm rotation angle, the tower arm posture, and the structural parameters of the construction site tower crane, wherein the camera is installed on the tower arm of the construction site tower crane; The camera posture is read according to a preset camera gimbal control system to determine the corresponding camera posture angle, and the corresponding real camera posture is determined according to the camera space position coordinates and the camera posture angle.

6. The method for dynamic 3D reconstruction of a large scene space according to claim 1, characterized in that: The performing of a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to the preset triangulation algorithm and the corrected camera pose to determine corresponding discrete three-dimensional point cloud data includes: Performing a matching operation on the construction site image key points in the adjacent video frame sequence to determine corresponding construction site image key point pairs; Performing a three-dimensional space triangulation operation based on the corrected camera pose and the key points of the construction site image to determine corresponding key point triangles, and performing a vertex solving operation on the key point triangles to determine corresponding vertex coordinates; A three-dimensional coordinate point recovery operation is performed according to the two-dimensional coordinates of the key point pairs of the construction site image and the vertex coordinates to determine corresponding discrete three-dimensional point cloud data.

7. The method for dynamic 3D reconstruction of a large scene space according to claim 1, characterized in that: The performing a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm to determine a corresponding three-dimensional model includes: Performing a gridding operation on the discrete three-dimensional point cloud data according to a preset triangulation algorithm to determine a corresponding three-dimensional grid; A mapping operation is performed on the three-dimensional grid according to texture information of the construction site image in the video frame sequence to determine a corresponding three-dimensional model.

8. A dynamic 3D reconstruction device for large scene spaces, characterized in that: The device comprises: A construction site image key point extraction module is configured to continuously shoot a target construction site using a preset camera to determine a corresponding video frame sequence, perform a target recognition operation on the video frame sequence according to a set target detection model, determine a specific target corresponding to each video frame, perform a key point extraction operation on the specific target to determine the corresponding key point, and perform a matching operation on multiple key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine the corresponding key point of the specific target, wherein the camera is mounted on the tower arm of a tower crane at the preset construction site to record the rotation of the tower arm; A preliminary camera pose determination module is configured to construct a key point pair based on the same specific target key point in adjacent video frames, construct a rotation matrix and a translation vector based on the key point pair, and determine the corresponding preliminary camera pose; a predicted camera pose determination module, configured to perform a reprojection error calculation operation on the key point pairs to determine a corresponding reprojection error, optimize the reprojection error according to a preset bundle adjustment algorithm, and update the preliminary camera pose according to a minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; The three-dimensional model construction module is used to perform a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, perform a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose, determine the corresponding discrete three-dimensional point cloud data, perform a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm, and determine the corresponding three-dimensional model.

9. A dynamic 3D reconstruction system for large scene spaces, characterized in that: The system includes a server and a preset camera communicatively connected to the server, wherein the server includes: A construction site image key point extraction module is configured to continuously shoot a target construction site using a preset camera to determine a corresponding video frame sequence, perform a target recognition operation on the video frame sequence according to a set target detection model, determine a specific target corresponding to each video frame, perform a key point extraction operation on the specific target to determine the corresponding key point, and perform a matching operation on multiple key points in adjacent video frames using a preset nearest neighbor matching algorithm to determine the corresponding key point of the specific target, wherein the camera is mounted on the tower arm of a tower crane at the preset construction site to record the rotation of the tower arm; A preliminary camera pose determination module is configured to construct a key point pair based on the same specific target key point in adjacent video frames, construct a rotation matrix and a translation vector based on the key point pair, and determine the corresponding preliminary camera pose; a predicted camera pose determination module, configured to perform a reprojection error calculation operation on the key point pairs to determine a corresponding reprojection error, optimize the reprojection error according to a preset bundle adjustment algorithm, and update the preliminary camera pose according to a minimum reprojection error obtained after the optimization operation to determine a corresponding predicted camera pose; The three-dimensional model construction module is used to perform a weighted fusion operation on the predicted camera pose and the set real camera pose to determine the corresponding corrected camera pose, perform a three-dimensional coordinate point recovery operation on the construction site image in the video frame sequence according to a preset triangulation algorithm and the corrected camera pose, determine the corresponding discrete three-dimensional point cloud data, perform a surface reconstruction operation on the discrete three-dimensional point cloud data according to a preset surface reconstruction algorithm, and determine the corresponding three-dimensional model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dynamic three-dimensional reconstruction method for a large scene space as described in any one of claims 1 to 7 are implemented.

Images