Construction scene-oriented multi-task synthetic data generation and three-dimensional reconstruction method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-15
- Publication Date
- 2026-08-11
Smart Images
Figure CN122550852A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a method for generating multi-task synthetic data and reconstructing three-dimensional structures for construction scenarios. Background Technology
[0002] With the widespread application of deep learning methods in construction safety monitoring and intelligent analysis, the lack of large-scale, high-quality labeled data has gradually become a major bottleneck restricting model performance and engineering applications. In real construction scenarios, complex conditions such as high-risk occlusion, extreme lighting, and dense multi-target interactions make it difficult to collect data on a large scale and in a controlled manner. Moreover, manually labeling information such as 3D location and depth maps is extremely time-consuming, labor-intensive, and costly. To address this issue, researchers have begun to explore methods for generating synthetic training data based on 3D virtual scene modeling and image rendering.
[0003] In existing research, the most common approach is to construct construction environments using game engines or 3D modeling software. By setting multiple perspectives, scales, and different operational states, labeled images are automatically generated and mixed with real data for model training. Studies have shown that this method can effectively expand sample size and distribution diversity, significantly improving object detection performance under conditions of small samples or class imbalance. To further reduce the difference between simulated data and real-world scenes, some studies have introduced domain randomization strategies, generating highly diverse data by systematically controlling lighting, texture, camera noise, and environmental interference factors. In recent years, some studies have begun to combine generative models for data augmentation, improving image realism while maintaining the consistency of automatic annotations.
[0004] However, existing research on synthetic construction data suffers from the following common problems. First, data generation often focuses on two-dimensional appearance changes, failing to adequately constrain the real geometric structure and spatial scale of the construction scene. This makes the generated data unsuitable for three-dimensional spatial location recovery or depth estimation tasks. Second, scene settings are primarily static or based on pre-defined behaviors, limiting the representation of continuous multi-target motion processes. The physical consistency between the simulation environment and the actual construction process is weak, failing to provide continuous three-dimensional trajectory data. Third, existing datasets typically serve only a single task (such as object detection), unable to simultaneously provide aligned data across multiple modalities, including RGB images, instance segmentation, accurate depth maps, and continuous three-dimensional motion trajectories. This restricts multi-task joint learning and the training of end-to-end three-dimensional vision models. Summary of the Invention
[0005] The purpose of this invention is to provide a method for generating multi-task synthetic data and reconstructing 3D in construction scenarios, which solves the problems of existing synthetic data methods for construction scenarios lacking realistic spatial geometric constraints and multi-target continuous motion representation, and being unable to generate multimodal alignment annotations simultaneously, resulting in insufficient training data and poor spatial perception capabilities for monocular 3D reconstruction models.
[0006] To achieve the above objectives, this invention provides a method for generating multi-task synthetic data and reconstructing three-dimensional structures for construction scenarios, comprising the following steps: Step 100: Referring to the construction video, construct a site geometric model in the virtual scene, build a construction target asset library and motion path, set two fixed monitoring positions with different spatial locations and at least two lighting conditions in the virtual scene, control the construction target to move according to the preset motion path under different conditions, render a preset number of color image sequences to form a geometric perception set, an occlusion perception set and a multi-target perception set, and simultaneously export the construction target category and bounding box annotation, and simultaneously export the depth map aligned on the pixel coordinates in the multi-target perception set. The value of the depth map represents the real distance from the target to the camera plane in the camera coordinate system; sample the preset motion path at fixed spatial intervals to obtain three-dimensional trajectory data; Step 200: Using YOLOv11 as the baseline model, replace the YOLOv11 feature extraction backbone with a feature extraction backbone composed of sequentially connected general inverted bottleneck modules, while preserving low-resolution, medium-resolution, and high-resolution feature maps in the backbone network. Figure 3 The multi-scale feature output interface is consistent with the baseline model; the target detection model is trained based on all color images and bounding box annotations. Step 300: Based on the Depth Anything v2 model, the Depth Anything v2 model is fine-tuned using a multi-target perception set and depth map as supervision to obtain a depth estimation model. Step 400: Input the color image at any time into the target detection model and the depth estimation model respectively. The target detection model outputs the construction target category and bounding box, and the depth estimation model outputs the depth map. Take the median of the depth map in the central sub-region of the bounding box as the representative depth of the construction target in the line of sight direction. Based on the known intrinsic parameter matrix composed of the camera focal length and principal point coordinates, back-project the center pixel coordinates of the bounding box to the camera coordinate system to obtain the three-dimensional spatial position of the construction target in the camera coordinate system, and combine them into a three-dimensional position sequence of the construction target according to the time step.
[0007] Furthermore, in step 100, the construction target asset library includes construction personnel models and construction machinery models. The construction machinery models include small loaders, loaders, excavators, concrete trucks, truck cranes, crawler cranes, and other vehicles. The top-down angles of the two fixed monitoring positions are both set to the same fixed top-down angle, both pointing to the scene operation area.
[0008] Furthermore, in step 100, the construction process of the geometric perception set, the occlusion perception set, and the multi-target perception set is as follows: Controlling a single construction target to move along a preset motion path, each virtual scene continuously renders a preset number of color image sequences to form a geometric perception set; Structural occlusions are introduced as fixed occlusions in each virtual scene to control the movement paths of multiple construction targets behind the structural occlusions without interfering with each other. Each virtual scene continuously renders a sequence of color images of a preset number of frames to form an occlusion perception set. The structural occlusions include upper tree occlusion, lateral tree occlusion, independent foundation structure occlusion, and slope structure occlusion. Multiple construction targets are simultaneously deployed in each virtual scene and made to move in parallel along a preset path. Each virtual scene continuously renders a preset number of color image sequences to form a multi-target perception set.
[0009] Further, in step 100, the method for generating three-dimensional trajectory data is as follows: sampling the preset motion path at equal spatial intervals to obtain discrete path points in the world coordinate system; establishing a time mapping based on the assumption of constant velocity motion of adjacent path points corresponding to adjacent time steps; and performing rigid body transformation on the path points through camera extrinsic parameters composed of rotation matrix and translation vector, so as to transform them from the world coordinate system to the camera coordinate system.
[0010] Further, in step 200, the general inverted bottleneck module includes a starting depth convolution, a channel expansion pointwise convolution, an intermediate depth convolution, and a channel compression pointwise convolution arranged sequentially along the signal flow direction; the feature extraction backbone is composed of the Stem module, the Stage3 module, the Stage4 module, and the Stage5 module in sequence, wherein the Stem module is composed of stepwise convolutions with a stride of two, used to compress the resolution of the input feature map twice; the Stage3 module, the Stage4 module, and the Stage5 module are all composed of several general inverted bottleneck modules, the first general inverted bottleneck module of each stage has a stride of two, used to complete the inter-stage downsampling, the stride of the remaining general inverted bottleneck modules is one, and the Stage5 module performs deep feature modeling by stacking general inverted bottleneck modules with a stride of one.
[0011] Furthermore, in step 200, the number of output channels of Stage3, Stage4 and Stage5 modules increases sequentially, and they correspond to multi-scale feature maps with step sizes of 8, 16 and 32, respectively; the feature fusion network at the back end of the backbone network maintains a bidirectional feature fusion structure from top to bottom and from bottom to top, and the detection head adopts a decoupled structure in which the classification branch and the regression branch are separated, and directly regresses the target center point and bounding box size based on the anchor box-free method.
[0012] Furthermore, in step 300, the method for fine-tuning the Depth Anythingv2 model using a multi-target perception set and depth map as supervision to obtain the depth estimation model includes: In the first phase, only the DPT decoder in the Depth Anything v2 model is unfrozen, and the depth decoding module is trained only using the multi-object perception set and depth map; In the second stage, based on the unfreezing in the first stage, the last two feature extraction layers of the DINOv2 encoder in the Depth Anything v2 model are also unfrozen, and the DPT decoder and the last two feature extraction layers are jointly trained using a multi-object perception set and depth map.
[0013] Furthermore, in step 400, the specific method for taking the median of the depth map within the central sub-region of the bounding box as the representative depth of the construction target in the line of sight is as follows: taking the center position of the bounding box as the center, expand to both sides in the horizontal and vertical directions respectively, with the expansion size being one-third of the width and height of the bounding box. The rectangular area formed by the expansion is taken as the central sub-region. The median of the depth map pixel values within the central sub-region is calculated, and the median is taken as the representative depth of the target in the line of sight.
[0014] Furthermore, in step 400, the formula for calculating the pixel coordinate back projection is as follows: ; in, and Represents the x and y coordinates of the pixel center of the bounding box; and Represents the x-coordinate and y-coordinate of the principal pixel; and This indicates the camera's focal length in the horizontal and vertical directions in pixels. This represents the median depth of the central subregion; and This represents the horizontal x-coordinate and horizontal y-coordinate of the target in the camera coordinate system.
[0015] Further, in step 400, the method of combining the three-dimensional position sequence of the construction target according to the time step is as follows: for each construction target, the three-dimensional spatial position of the construction target at each time step is spliced together in time order to obtain a three-dimensional position sequence; the three-dimensional position sequence is represented by the camera coordinate system, and the time interval between adjacent time steps is consistent with the time step interval of the three-dimensional trajectory data.
[0016] Therefore, the present invention employs the above-mentioned multi-task synthetic data generation and 3D reconstruction method for construction scenarios, which has the following beneficial effects: 1. By generating multi-task synthetic samples with spatial scale and temporal alignment characteristics by functional subsets in virtual scenes, the problem of data appearance enhancement and 3D position supervision being mutually separated in the existing technology is solved, so that the same batch of samples can simultaneously support the training and validation of target detection, monocular depth estimation and 3D position reconstruction tasks. 2. By introducing a general inverted bottleneck module to reconstruct the YOLOv11 backbone, the model parameter size and memory usage are significantly reduced while keeping the multi-scale feature interface unchanged, thus improving the inference efficiency of the detection network under edge computing conditions. 3. By employing a two-stage strategy of freezing and partially thawing, the monocular depth estimation model is scale-aligned and detail-refined from the perspective of construction monitoring. This alleviates the problems of scale compression and long-distance structural blurring in the original model under large depth-of-field overhead conditions, enabling the depth map to directly correspond to the true distance in the camera coordinate system.
[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0018] Figure 1 This is a flowchart of a multi-task synthetic data generation and 3D reconstruction method for construction scenarios according to the present invention; Figure 2 This is a schematic diagram of the construction site monitoring images and scene elements of the present invention; Figure 3 This is a schematic diagram of the virtual construction scene layout and structural obstructions of the present invention; Figure 4 This is a schematic diagram of the spatial distribution of the preset movement path in the construction scenario of the present invention; Figure 5 This is a schematic diagram of a sample of the geometric perception set of the present invention; Figure 6 This is a schematic diagram of the occlusion perception set of the present invention; Figure 7 This is a schematic diagram of the network structure of YOLOv11 of the present invention; Figure 8 This is a schematic diagram of the structure of the YOLOv11 target detection model based on UIB improved according to the present invention; Figure 9 This is a schematic diagram showing the changes in various indicators during the training process of the target detection model of the present invention; Figure 10 This is a schematic diagram showing the changes in various fine-tuning indicators of the depth estimation model after targeted fine-tuning of the Depth Anything v2 model of the present invention. Figure 11 This is an example diagram of the monocular depth estimation results of the present invention. Detailed Implementation
[0019] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely illustrates selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0020] Please see Figure 1 A method for multi-task synthetic data generation and 3D reconstruction for construction scenarios includes the following steps: Step 100: Referring to the construction video, construct a site geometric model in the virtual scene, build a construction target asset library and motion path, set two fixed monitoring positions with different spatial locations and at least two lighting conditions in the virtual scene, control the construction target to move according to the preset motion path under different conditions, render a preset number of color image sequences to form a geometric perception set, an occlusion perception set, and a multi-target perception set, and simultaneously export the construction target category and bounding box annotation, and simultaneously export the depth map aligned on the pixel coordinates from the multi-target perception set. The value of the depth map represents the real distance from the construction target to the camera plane in the camera coordinate system; sample the preset motion path at fixed spatial intervals to obtain three-dimensional trajectory data, specifically including the following sub-steps: Step 101, Virtual Scene Construction; like Figure 2-3 As shown, using video footage of a real construction site as a reference, the main roads, work areas, and structural obstructions that significantly affect the visibility of construction targets are identified and abstracted in a virtual scene. These structural obstructions include, but are not limited to, pillars, material piles, and trees. During the modeling process, the source of occlusion in the virtual scene is limited to structural occlusion. The position and scale parameters of each structural obstruction are parameterized and recorded during the modeling phase to facilitate the controllable construction of the occlusion state along the same path in subsequent occlusion perception sets.
[0021] Step 102, Configure the construction target asset library; Construction targets are placed within the constructed virtual scene. The construction target asset library includes models of construction personnel and construction machinery. The machinery models include small loaders, loaders, excavators, concrete trucks, truck cranes, crawler cranes, and other vehicles. Each construction target is configured with one or two model instances with different geometric appearances and surface textures. Each model instance has its typical geometric dimensions pre-defined. The construction target asset library constructed in this embodiment is shown in Table 1. Table 1 Configuration of Construction Target Asset Pool
[0022] Maintaining differences in geometry and texture among similar construction targets can reduce overfitting of subsequent detection models to a single geometric shape or texture, thereby improving the model's generalization ability in real and virtual scenes.
[0023] Step 103, setting observation conditions; Two fixed monitoring positions, denoted as Position A and Position B, are set up in the virtual scene. Both positions have the same top-down angle, which is set to 20 degrees in this embodiment, and a focal length of 30 mm. The visual center of both positions points to the core working area of the virtual scene, with coordinates of (7 m, 7 m, 30 m) and (135 m, 5 m, 25 m) respectively, and the visual center coordinates of (80 m, 80 m, 0 m). While keeping the spatial layout and movement trajectory of the construction targets unchanged, two representative lighting conditions are introduced, corresponding to the brightness distribution of the construction site under different time periods or weather conditions. Under the same top-down angle, different scale distributions of construction targets can be naturally formed on the image plane by changing the spatial position of the camera without introducing viewpoint disturbance; the lighting condition disturbance only acts on the imaging layer without changing the geometry and motion semantics of the virtual scene, thereby decoupling scale changes, lighting changes, and occlusion changes, which facilitates subsequent evaluation of the impact of each factor on the detection and depth estimation models.
[0024] Step 104, setting the motion path; like Figure 4 As shown, this embodiment combines the functional zoning of the construction site and the work organization method to identify and preset fourteen movement paths for construction targets in a virtual scene. These movement paths are divided into five areas: the central passage, the southern work area, the northern work area, the western passage, and the eastern work area. The spatial attributes of each route are shown in Table 2. Table 2 Spatial Attributes of Typical Movement Routes in Construction Scenarios
[0025] Abstracting the movement of construction targets based on actual construction processes can avoid completely random trajectories from deviating from construction semantics, making the synthesized trajectory representative in terms of spatial distribution and movement patterns. Furthermore, the combination of short-distance local movement and long-distance cross-regional movement can cover two typical risk virtual scenarios: close-range interaction and long-distance approach.
[0026] Step 105, Generation of the geometric perception set; like Figure 5As shown, the geometry perception set is used to support the model's learning of geometric features of a single construction target under multi-view and multi-scale conditions. A single construction target independent display strategy is adopted, meaning that only one construction target model is placed in the virtual scene at a time, and other construction targets do not appear. In this embodiment, for each construction target model in the construction target asset library, it is first displayed on the turning operation route in the northern region to collect close-range multi-angle geometric features. Then, two to three routes are selected from the remaining unobstructed routes for motion display of the construction target. In this embodiment, each construction target continuously renders 50 frames of image sequences on each selected route, and the corresponding bounding box annotations are simultaneously exported. Combining two fixed monitoring positions and two lighting conditions, four imaging conditions can be formed, quadrupling the number of samples generated for each route. A total of 184 virtual scene sequences, totaling 9200 frames of color images, are included as the geometry perception set. The construction target and route configuration of the geometry perception set is shown in Table 3. Table 3. Construction objectives and movement route configuration of the geometric perception set
[0027] Under pure observation conditions that exclude multi-target occlusion and interference, the model can focus on the stable feature representation of the construction target under scale and attitude changes, providing a unified geometric reference for subsequent occlusion perception sets and multi-target perception sets.
[0028] Step 106, occlusion perception set generation; Occlusion perception sets are used to characterize the visibility changes of construction targets under structural occlusion conditions. For example... Figure 6 As shown, under the premise of ensuring that the spatial distribution of each route does not interfere with each other, at least five routes with significant occlusion characteristics are selected. These routes cover four typical occlusion forms: upper tree occlusion, lateral tree occlusion, independent foundation structure occlusion, and slope structure occlusion. This step adopts a multi-target simultaneous display strategy, where three construction target models are simultaneously deployed in each virtual scene. The construction targets are displayed sequentially along the selected routes. Different virtual scenes form diverse occlusion pattern combinations by changing the combination and order of construction targets. Simultaneously, by increasing the height of independent foundation structures, the unoccluded state evolves into structural occlusion of the construction targets under the same path conditions. In this embodiment, each occlusion virtual scene continuously renders 50 frames of image sequences along the corresponding route and simultaneously exports bounding box annotations. Combining two camera positions and two lighting conditions, a total of 20 occlusion virtual scenes corresponding to 80 virtual scene sequences are constructed, totaling four thousand frames of color images as the occlusion perception set. The construction target and route configuration of the occlusion perception set is shown in Table 4. Table 4 Construction objectives and route configuration for occlusion sensing sets
[0029] By reusing some paths from the geometric perception set and dynamically generating structural occlusions on the same path, structural occlusion morphology with evolutionary characteristics is introduced, making occlusion factors the main variable affecting the visibility of construction targets. This allows for the controllable evaluation of the model's feature degradation and perception robustness under occlusion conditions.
[0030] Step 107, Generation of multi-target perception set; The multi-target perception set is used to characterize the complex perception conditions under the parallel operation of multiple targets at the construction site. This step constructs 12 virtual scenes. Each virtual scene arranges and organizes the movement of construction targets according to the actual construction steps. Five to six construction targets are placed simultaneously in each virtual scene, covering construction personnel and various types of construction machinery. The timing, spatial location, and movement path of different construction targets are set according to the construction process. Construction targets may move simultaneously along the same or adjacent routes, forming a virtual scene layout of multiple targets moving in parallel within multiple work channels. Four shooting conditions are formed by combining two camera positions and two lighting conditions. In this embodiment, each virtual scene continuously renders 100 frames of image sequences. A total of 48 virtual scene sequences are formed, totaling 4,800 color images as the multi-target perception set. Simultaneously, a depth map, strictly aligned with the color images in pixel coordinates, is directly exported from the depth channel in the virtual rendering pipeline within the multi-target perception set. The depth map value represents the true distance from the construction target to the camera plane in the camera coordinate system and is unaffected by sensor noise. A total depth map is generated. Figure 4 1,800 images. The construction targets and route configurations of the multi-target perception set are shown in Table 5: Table 5. Construction targets and route configuration of the multi-target perception set
[0031] Frequent occlusion switching and visibility changes in multi-target virtual scenes can provide supervision signals that closely resemble real construction monitoring conditions for depth estimation models. Furthermore, pixel-level aligned real depth maps are more suitable as fine-tuning supervision for absolute depth estimation models compared to sparse depth maps acquired in reality.
[0032] Step 108: Generation of 3D trajectory data.
[0033] Input a predefined motion path, and perform discrete sampling along the path arc length at equidistant spatial intervals to obtain discrete path points in the world coordinate system. Establish construction target. The corresponding path curve is obtained after discrete sampling. The spatial trajectory of each path point in the world coordinate system is represented as follows: ; in, Indicates construction objectives In the The spatial location of each path sampling point in the world coordinate system; , , Representing the construction objectives In the Each path sampling point is located in the coordinate components of three directions in the world coordinate system; This represents the total number of discrete path points obtained by sampling along this path.
[0034] To ensure geometric consistency between trajectory data and the image observation reference frame, the path points in the world coordinate system are rotated using the camera's rotation matrix. With translation vector Perform a rigid body transformation to obtain the 3D position in the camera coordinate system: ; in, Indicates construction objectives In the The three-dimensional position of each path sampling point in the camera coordinate system; This represents the rotation matrix from the world coordinate system to the camera coordinate system obtained from the camera's extrinsic parameters; This represents the translation vector from the world coordinate system to the camera coordinate system, obtained from the camera's extrinsic parameters.
[0035] Since path points are sampled at fixed spatial intervals, this step further introduces the assumption of approximately constant velocity motion, mapping adjacent path points to adjacent time steps to obtain the construction target. The position trajectory that changes over time: ; in, Indicates construction objectives In the Each time step is located in the three-dimensional position in the camera coordinate system; Indicates construction objectives 3D trajectory data; , , Representing the construction objectives In the Each time step is defined by the horizontal x-coordinate, horizontal y-coordinate, and line-of-sight depth in the camera coordinate system. Complete trajectories are generated from two monitoring positions, and sliding windows are used to extract data along the time dimension at fixed steps based on equidistant sampling. This embodiment ultimately yields a total of 3112 sets of 3D trajectory data, with each sample maintaining consistency in time step definition, spatial scale, and coordinate representation. Through the combined effect of equidistant sampling and isokinetic time mapping, the spatial distribution and time intervals of the trajectory data have clear physical meanings and can serve as a true reference for subsequent verification of the accuracy of 3D position reconstruction and trajectory prediction.
[0036] Step 100 outputs a multi-task synthetic dataset integrating the construction target detection task, monocular depth estimation task, and 3D trajectory analysis task. The construction target detection task corresponds to a target detection dataset composed of a geometric perception set, an occlusion perception set, a multi-target perception set, and corresponding bounding box annotations. The monocular depth estimation task corresponds to a depth estimation dataset composed of a multi-target perception set and its corresponding depth map. The 3D trajectory analysis task corresponds to a trajectory prediction dataset composed of 3D trajectory data. The output of this multi-task dataset serves as the input for model training in steps 200 and 300, and as the ground truth reference for the 3D position reconstruction results in step 400, ensuring consistency in coordinate system, time step, and construction target identification among subsequent steps.
[0037] Step 200: Using YOLOv11 as the baseline model, replace the YOLOv11 feature extraction backbone with a feature extraction backbone composed of sequentially connected general inverted bottleneck modules, while preserving low-resolution, medium-resolution, and high-resolution feature maps in the backbone network. Figure 3 The multi-scale feature output interface is consistent with the baseline model; the target detection model is trained based on all color images and bounding box annotations, specifically including the following sub-steps: Step 201: Baseline boundary framework determined; YOLOv11 was selected as the baseline model; in this embodiment, the YOLOv11-nano version was specifically used. Figure 7 As shown, the baseline model consists of three parts: a backbone network, a feature fusion network, and a head. The backbone network is formed by connecting several cross-layer spliced feature extraction modules, with a spatial pyramid fast pooling module at the end of the backbone to expand the receptive field. The feature fusion network adopts a bidirectional feature fusion structure combining a top-down feature pyramid path and a bottom-up path aggregation network. The head adopts a decoupled structure with separate classification and regression branches, and directly regresses the center point and bounding box size of the construction target based on an anchor-free method. This baseline model provides a clear multi-scale feature interface for subsequent backbone reconstruction, ensuring that backbone replacement does not affect subsequent feature fusion and head modules.
[0038] Step 202, Object detection model construction; like Figure 8 As shown, a target detection model is constructed by replacing the cross-layer concatenation feature extraction module in the baseline model with a general inverted bottleneck module as a lightweight backbone network. The structure of the general inverted bottleneck module includes, sequentially arranged along the signal flow direction: optional enabled initial depth convolution, channel expansion pointwise convolution, optional enabled intermediate depth convolution, and channel compression pointwise convolution. The lightweight backbone network consists of the Stem module, Stage3 module, Stage4 module, and Stage5 module in sequence. The specific composition is as follows: (1) The Stem module consists of two concatenated convolutional layers with a stride of 2, which are used to continuously downsample the resolution of the input image twice to obtain the P1 feature layer and the P2 feature layer. (2) The Stage3 module consists of two general inverted bottleneck modules connected in series. The first module has a stride of 2 and simultaneously enables 3x3 initial depth convolution and 5x5 intermediate depth convolution to complete the downsampling from the P2 feature layer to the P3 feature layer. The second module has a stride of 1 and only enables 3x3 intermediate depth convolution. The channel expansion ratio is uniformly set to 2.5 and the number of channels is 64. (3) The Stage4 module consists of three general inverted bottleneck modules connected in series. The first module has a stride of two and simultaneously enables three-by-three initial depth convolution and five-by-five intermediate depth convolution to complete the downsampling from the P3 feature layer to the P4 feature layer. The other two modules have a stride of one and only enable three-by-three intermediate depth convolution. The channel expansion ratio is uniformly set to 2.5, and the number of channels is 128. (4) The Stage5 module consists of several general inverted bottleneck modules connected in series. In this embodiment, five modules are stacked. The first module has a stride of two and simultaneously enables three-by-three initial depth convolution and five-by-five intermediate depth convolution. The remaining modules have a stride of 1 and only enable three-by-three intermediate depth convolution. The channel expansion ratio is uniformly set to 2.5, and the number of channels is 256.
[0039] After the above structural reconstruction, the lightweight backbone network outputs multi-scale feature maps with strides of 8, 16, and 32, which are fully aligned with the multi-scale feature interface of the baseline model. Therefore, the feature fusion network and the detection head structure can be directly reused without modification. Compared to the cross-layer splicing module, which needs to save multiple sets of intermediate features at each stage before splicing, the general inverted bottleneck module only completes channel expansion and compression along a single path, resulting in lower memory access costs. At the same time, by distributing spatial modeling capabilities into the initial depthwise convolution and intermediate depthwise convolution, they can be selectively enabled according to the feature scale, thereby improving spatial context modeling capabilities without significantly increasing the parameter size. The output of this step is the object detection model, whose parameter count is reduced from 2,591,400 in the original baseline to 1,081,788, a reduction of approximately 58.25%.
[0040] Step 203, Preprocessing of training samples; First, the data is divided into training and validation sets at an 8:2 ratio. During this division, it is ensured that consecutive frames from the same virtual scene do not appear simultaneously in both the training and validation sets to avoid data leakage caused by temporal correlation. Then, the input images are uniformly scaled using a long-side aligned, proportional scaling method, with the construction target size set to 640×640. Short side blanks are padded to prevent deformation caused by non-proportional scaling. Pixel values are normalized to the [0,1] range, and the channel order is adjusted according to the boundary frame requirements. Annotation information is synchronized with the image to ensure consistency in the position and size of the construction target bounding box after data augmentation. Extremely small construction targets that may appear after scale changes or cropping are filtered based on a minimum side length of 8 pixels to avoid invalid annotations interfering with model training stability. During the training phase, additional data augmentation methods such as random horizontal flipping and random scale scaling are introduced to enhance the model's robustness to changes in the pose and scale of the construction targets. No data augmentation is used on the validation set to ensure the objectivity of model performance evaluation.
[0041] Step 204, Model Training.
[0042] The object detection model was trained using stochastic gradient descent as the optimizer, with an initial learning rate of 0.01 and a final learning rate of 0.0001. A linear decay learning rate strategy was employed, with a weight decay of 0.0005. The batch size was 32, and the training run consisted of 200 epochs, with the input size maintained at 640×640. A three-epoch warm-up process was implemented at the beginning of training to gradually increase the learning rate from a small value, thereby improving convergence stability. During training, automatic mixed precision was enabled to reduce memory usage and improve computational efficiency. In the last ten epochs, multi-image stitching data augmentation was disabled to reduce data distribution disturbances and improve the model's later convergence quality.
[0043] The changes in metrics during the training process of the object detection model are as follows: Figure 9 As shown, the optimization stability of the model is analyzed from the loss function change curves. During the training phase, the bounding box regression loss (train / box loss) rapidly decreased from an initial value of approximately 4.35, stabilizing at around 0.5 after about 170 epochs; the classification loss (train / cls loss) decreased from 7.09 to around 0.3, with a similar change process to the bounding box regression loss; the distribution focus loss (train / dfl loss) also showed a continuous decreasing trend, entering a slow optimization phase in the mid-to-late stages. The validation set loss curve maintained the same trend as the training set loss, with the bounding box regression loss eventually stabilizing at around 0.54, and the difference between it and the training loss being approximately Δ=0.047, within a reasonable range; the decrease rates of the classification loss and distribution focus loss were basically synchronized with the training phase, and no significant oscillations or divergences occurred throughout the training process. The co-convergence of the training loss and validation loss indicates that the model has good generalization ability during the optimization process and no significant overfitting occurred. After introducing the UIB lightweight backbone structure, the model still maintains stable training characteristics and convergence behavior.
[0044] After training, the performance of the object detection model was primarily evaluated using precision, recall, and average precision (AP). mAP 0.5 represents the detection performance at an IoU threshold of 0.5, while mAP 0.5-0.95 provides a comprehensive evaluation of the model's localization accuracy within a more stringent multi-threshold range. The precision curve shows a rapid increase in the initial training phase, stabilizing at around 96% after approximately 50 training epochs, indicating that the model can develop effective object discrimination capabilities within a relatively small number of training epochs. The recall curve exhibits a gradual upward trend, eventually stabilizing at approximately 94%, demonstrating that as the feature representation capability continuously improves, the model's ability to detect objects at different scales gradually increases. The synergistic increase in precision and recall indicates that the model effectively improves object recall while reducing false positives, achieving a dynamic balance between the two. The mAP of 0.5 rises rapidly in the early stages of training, reaching approximately 0.94, and then enters a stable phase. In contrast, the improvement process of mAP from 0.5 to 0.95 is more gradual, eventually converging to approximately 0.83, indicating that the model continuously optimizes the bounding box localization accuracy during training. The performance and inference efficiency comparisons between the baseline model and the object detection model are shown in Tables 6 and 7. Table 6 Performance Comparison between Baseline Model and Object Detection Model
[0045] Table 7 Comparison of inference efficiency between baseline model and object detection model
[0046] Under the same hardware environment and input size (640×640), the YOLO11n model has 2.59M parameters, an average inference time of 14.09ms per frame (70.96 FPS), and a GPU memory footprint of 52.0MB. After adopting the UIB lightweight backbone, the object detection model's parameter count is reduced to 1.08M, a reduction of approximately 58.25%; the average inference time is reduced to 11.19ms, corresponding to 89.36 FPS, representing an improvement in inference speed of approximately 25.9%; and the GPU memory footprint is reduced to 42.1MB. The results demonstrate that the lightweight backbone structure effectively reduces the model's computational and storage overhead, thereby improving its operational efficiency.
[0047] Step 300: Based on the Depth Anything v2 model, the Depth Anything v2 model is fine-tuned using a multi-target perception set and depth map as supervision to obtain a depth estimation model. This embodiment selects a monocular depth estimation model supporting absolute depth estimation as the starting point for fine-tuning. The Depth Anything v2 (DA2) model is chosen as the foundation. The Depth Anything v2 model employs a Transformer-based encoder-decoder structure, including an input layer, an image embedding module, a DINOv2 encoder, a DPT decoder, and an output layer. The DINOv2 encoder extracts global semantic and geometric features using DINOv2 and contains multiple stacked feature extraction layers. The DPT decoder recovers a high-resolution depth map through multi-scale feature fusion, achieving a fine representation of complex scene structures. Simultaneously, DA2 provides an absolute depth estimation mode, directly outputting depth results in meters, meeting the spatial computing needs of engineering applications. However, the training data for the publicly available weights of this model mainly comes from indoor environments or road scenes, with depth distribution typically concentrated within 0–80m. Construction monitoring scenes, however, exhibit significantly different imaging characteristics. Under high-position fixed camera conditions, the target activity area is mainly distributed within the 0–150m range, and the ground plane occupies a large proportion of the scene with a clear overhead view, resulting in a deviation in the depth statistical distribution from the original training data. Using a pre-trained model directly can easily lead to scale compression or structural smoothing in long-distance regions, affecting the accuracy of spatial distance calculations. Therefore, while preserving the geometric expressive power of the pre-trained model, the DA2 absolute depth estimation model needs to be fine-tuned. This fine-tuning includes: The first stage employs a freeze strategy, unfreezing only the DPT decoder of the DA2 model and training only the DPT decoder using the depth estimation dataset to quickly achieve stable alignment of the output scale. The second stage employs a partial unfreezing strategy, unfreezing the last two feature extraction layers of the DINOv2 encoder based on the first stage for training, and using the depth estimation dataset to jointly train the DPT decoder and the unfrozen last two feature extraction layers to enhance the model's ability to express scene textures and boundary details.
[0048] Table 8 shows the targeted fine-tuning training settings for the Depth Anything v2 model: Table 8. Targeted Fine-tuning Training Settings for the Depth Anything v2 Model
[0049] Similar to the performance evaluation metrics for object detection models, the performance of the depth estimation model obtained after targeted fine-tuning of the Depth Anything v2 model is evaluated. The changes in its training and validation metrics are as follows: Figure 10 As shown in the figure. From the performance metrics, the absolute relative error and the squared relative error converged to approximately 0.126 and 1.86 respectively in the later stages of training. The δ precision index continued to improve, with δ1 / δ2 / δ3 reaching 0.794 / 0.988 / 0.999, significantly improving the consistency between the predicted depth and the actual depth. The RMSE decreased from 9.808 at the beginning of training to 8.98, indicating that the model still has some bias in long-distance regions, which is related to the inherent uncertainty of monocular vision in large-scale scenes. Considering the actual needs of construction risk analysis, it can be found that human-computer interaction risks are mainly concentrated in the medium to short distance range. Within this distance range, the model maintains good numerical consistency between the predicted depth and the actual depth, which can meet the needs of subsequent 3D position calculation.
[0050] The actual distance range of the construction site is 0-140m, and the depth estimation results are shown in the example below. Figure 11 As shown, the model clearly distinguishes between foreground objects and distant background areas, and personnel, construction machinery, and site structures exhibit a clear hierarchical relationship in the depth map. Even in areas with structural occlusion, the depth prediction maintains good continuity without large-area breaks or abnormal jumps. These results demonstrate that, after fine-tuning with construction scene data, Depth Anything v2 is capable of representing spatial structures under large depth-of-field overhead viewing conditions.
[0051] Through targeted training on the depth estimation dataset, the depth estimation module achieves recalibration of the distribution at long-distance scales while maintaining global structural consistency, providing a stable and physically meaningful spatial input for subsequent target 3D position mapping and human-machine collision risk calculation.
[0052] Step 400: Input the color image at any time into the target detection model and the depth estimation model respectively. The target detection model outputs the construction target category and bounding box, and the depth estimation model outputs a depth map. Take the median of the depth map within the central sub-region of the bounding box as the representative depth of the construction target in the line of sight direction. Based on the known intrinsic parameter matrix composed of the camera focal length and principal point coordinates, back-project the center pixel coordinates of the bounding box to the camera coordinate system to obtain the three-dimensional spatial position of the construction target in the camera coordinate system, and combine them into a three-dimensional position sequence of the construction target according to the time steps. Specifically, it includes the following sub-steps: Step 401 represents depth extraction; For any time (the... Using color images (at each time step) as input, the detection network detects the construction target. Output two-dimensional bounding box The bounding box is determined by the center pixel coordinates of the bounding box. , bounding box width and bounding box height Four parameters describe, namely Simultaneously, the depth estimation model outputs a depth map that is pixel-level aligned with the color image. To reduce the impact of bounding box boundary errors and local noise on depth acquisition, the bounding box is expanded horizontally and vertically to both sides, with the expansion dimension being one-third of the bounding box width and height. The resulting rectangular region is designated as the central sub-region of the bounding box, denoted as […]. The median of the depth values of all pixels within this sub-region is used as the representative depth of the construction target, specifically satisfying the following relationship: ; in, Indicates construction objectives In the The representative depth at each time step in the direction of the camera's line of sight; Representing a depth map In pixel coordinates The depth value at that location; This represents a rectangular sub-region centered at the bounding box, with dimensions one-third of the bounding box's width and height. Using the median instead of the mean suppresses extreme value interference from boundary and background pixels, and taking the center-third sub-region ensures that sampling points stably fall on the construction target, thus improving the stability of the depth representation.
[0053] Step 402, pixel back projection calculation. Let the camera intrinsic parameter matrix be... Composed of the camera focal length and the principal point coordinates, specifically in the following form: ; in, and This indicates the camera's focal length in the horizontal and vertical directions in pixels. and Represents the x-coordinate and y-coordinate of the principal pixel; This represents the camera intrinsic parameter matrix. In this embodiment, it is calculated based on the camera focal length and sensor size from the animation software: ; ; ; Camera intrinsics can be calculated through pre-calibration using the focal length and sensor size of a virtual scene or a real camera. Based on representative depth... Set the center pixel coordinates of the bounding box By back-projecting onto the camera coordinate system, the construction target can be obtained. The formula for calculating the 3D spatial position in the camera coordinate system is as follows: ; ; ; in, , and Representing the construction objectives In the Each time step is defined by the horizontal x-coordinate, horizontal y-coordinate, and depth along the line of sight in the camera coordinate system. Indicates construction objectives In the Each time step represents a 3D spatial position in the camera coordinate system. In the camera coordinate system, the positive direction of the third axis is the forward line of sight, the positive direction of the first axis is the horizontal rightward direction of the image, and the positive direction of the second axis is the vertical upward direction of the image. After mapping the pixel coordinates to the camera coordinate system using the back projection formula, the resulting 3D position is completely consistent in scale with the true 3D position obtained through extrinsic parameter transformation in step 108. This ensures that the observation reconstruction result and the true trajectory are in the same geometric space and can be directly used for accuracy evaluation.
[0054] Step 403, Construction of the three-dimensional position sequence.
[0055] For each time step, all detected construction targets in the virtual scene undergo representative depth extraction in step 401 and pixel back projection calculation in step 402 to obtain the set of three-dimensional spatial positions of all construction targets in the virtual scene at that time step; the same construction target The three-dimensional spatial positions at continuous time steps are stitched together in chronological order to form a three-dimensional position sequence of the construction target: ; in, This represents the number of time steps corresponding to the total duration of the observation sequence; Indicates construction objectives In the The three-dimensional spatial location at each time step. Within the same construction target. Between different time steps, the construction target identifiers output by the construction target detection network are used for association to ensure the consistency of construction targets in the continuous time dimension. The output of this step is a three-dimensional position sequence at the construction target level. Its coordinate system definition and time step length are completely consistent with the three-dimensional trajectory data in step 108, and it can be directly used as input for subsequent trajectory prediction.
[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for multi-task synthetic data generation and 3D reconstruction for construction scene, characterized in that, Includes the following steps: Step 100: Referring to the construction video, construct a site geometric model in the virtual scene, build a construction target asset library and movement path, set two fixed monitoring positions with different spatial locations and at least two lighting conditions in the virtual scene, control the construction target to move according to the preset movement path under different conditions, render a preset number of color image sequences to form a geometric perception set, an occlusion perception set and a multi-target perception set, and simultaneously export the construction target category and bounding box annotation, and simultaneously export the depth map aligned on the pixel coordinates in the multi-target perception set. The value of the depth map represents the real distance from the target to the camera plane in the camera coordinate system. Three-dimensional trajectory data is obtained by sampling a preset motion path at fixed spatial intervals; Step 200: Using YOLOv11 as the baseline model, replace the YOLOv11 feature extraction backbone with a feature extraction backbone composed of sequentially connected general inverted bottleneck modules. In the backbone network, the three-layer multi-scale feature output interface of low-resolution feature map, medium-resolution feature map and high-resolution feature map is kept consistent with the baseline model. The target detection model is trained based on all color images and bounding box annotations; Step 300: Based on the Depth Anything v2 model, the Depth Anything v2 model is fine-tuned using a multi-target perception set and depth map as supervision to obtain a depth estimation model. Step 400: Input the color image at any time into the target detection model and the depth estimation model respectively. The target detection model outputs the construction target category and bounding box, and the depth estimation model outputs the depth map. Take the median of the depth map in the central sub-region of the bounding box as the representative depth of the construction target in the line of sight direction. Based on the known intrinsic parameter matrix composed of the camera focal length and principal point coordinates, back-project the center pixel coordinates of the bounding box to the camera coordinate system to obtain the three-dimensional spatial position of the construction target in the camera coordinate system, and combine them into a three-dimensional position sequence of the construction target according to the time step.
2. The method of claim 1, wherein, In step 100, the construction target asset library includes construction personnel models and construction machinery models. The construction machinery models include small loaders, loaders, excavators, concrete trucks, truck cranes, crawler cranes, and other vehicles. The top-down angles of the two fixed monitoring positions are both set to the same fixed top-down angle, both pointing to the scene operation area.
3. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 2, characterized in that, In step 100, the construction process of the geometric perception set, the occlusion perception set, and the multi-target perception set is as follows: Controlling a single construction target to move along a preset motion path, each virtual scene continuously renders a preset number of color image sequences to form a geometric perception set; In each virtual scene, structural occlusions are introduced as fixed occlusions to control the movement paths of multiple construction targets behind the structural occlusions without interfering with each other. Each virtual scene continuously renders a sequence of color images of a preset number of frames to form an occlusion perception set. The structural occlusions include upper tree occlusion, lateral tree occlusion, independent foundation structure occlusion, and slope structure occlusion. Multiple construction targets are simultaneously deployed in each virtual scene and made to move in parallel along a preset path. Each virtual scene continuously renders a preset number of color image sequences to form a multi-target perception set.
4. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 3, characterized in that, In step 100, the method for generating three-dimensional trajectory data is as follows: sampling the preset motion path at equal spatial intervals to obtain discrete path points in the world coordinate system; establishing a time mapping based on the assumption of constant velocity motion of adjacent path points corresponding to adjacent time steps; and performing rigid body transformation on the path points through camera extrinsic parameters composed of rotation matrix and translation vector to transform them from the world coordinate system to the camera coordinate system.
5. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 4, characterized in that, In step 200, the general inverted bottleneck module includes a starting depth convolution, channel expansion pointwise convolution, intermediate depth convolution and channel compression pointwise convolution arranged sequentially along the signal flow direction; the feature extraction backbone is composed of the Stem module, Stage3 module, Stage4 module and Stage5 module in sequence, wherein the Stem module is composed of stepwise convolution with a stride of two, which is used to compress the resolution of the input feature map twice; The Stage3, Stage4, and Stage5 modules are all composed of several general inverted bottleneck modules. The first general inverted bottleneck module in each stage has a step size of two, which is used to complete the inter-stage downsampling. The step size of the remaining general inverted bottleneck modules is one. The Stage5 module performs deep feature modeling by stacking general inverted bottleneck modules with a step size of one.
6. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 5, characterized in that, In step 200, the number of output channels of Stage3, Stage4 and Stage5 modules increases sequentially, and they correspond to multi-scale feature maps with step sizes of 8, 16 and 32 respectively. The feature fusion network at the back end of the backbone network maintains a bidirectional feature fusion structure from top to bottom and bottom to top. The detection head adopts a decoupled structure with separate classification and regression branches, and directly regresses the target center point and bounding box size based on the anchorless method.
7. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 6, characterized in that, In step 300, the method for fine-tuning the Depth Anything v2 model using a multi-target perception set and depth map as supervision to obtain the depth estimation model includes: In the first phase, only the DPT decoder in the Depth Anything v2 model is unfrozen, and the depth decoding module is trained only using the multi-object perception set and depth map; In the second stage, based on the unfreezing in the first stage, the last two feature extraction layers of the DINOv2 encoder in the Depth Anything v2 model are also unfrozen, and the DPT decoder and the last two feature extraction layers are jointly trained using a multi-object perception set and depth map.
8. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 7, characterized in that, In step 400, the specific method for taking the median of the depth map in the central sub-region of the bounding box as the representative depth of the construction target in the line of sight is as follows: taking the center position of the bounding box as the center, expand to both sides in the horizontal and vertical directions respectively. The expansion size is one-third of the width and height of the bounding box. The rectangular area formed by the expansion is taken as the central sub-region. The median of the depth map pixel values in the central sub-region is taken as the representative depth of the target in the line of sight.
9. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 8, characterized in that, In step 400, the formula for calculating the pixel coordinate back projection is as follows: ； in, and Represents the x and y coordinates of the pixel center of the bounding box; and Represents the x-coordinate and y-coordinate of the principal pixel; and This indicates the camera's focal length in the horizontal and vertical directions in pixels. This represents the median depth of the central subregion; and This represents the horizontal x-coordinate and horizontal y-coordinate of the target in the camera coordinate system.
10. The method for multi-task synthetic data generation and 3D reconstruction for construction scenarios according to claim 9, characterized in that, In step 400, the method for combining the three-dimensional position sequence of the construction target according to the time step is as follows: for each construction target, the three-dimensional spatial position of the construction target at each time step is spliced together in time order to obtain the three-dimensional position sequence; the three-dimensional position sequence is represented by the camera coordinate system, and the time interval between adjacent time steps is consistent with the time step interval of the three-dimensional trajectory data.