A general semantic mapping method for tower cranes in architectural scenes

By acquiring images from a monocular RGB camera at the boom end of a tower crane and combining them with a multimodal large model and tower crane slewing scale calibration, semantic maps and point cloud maps of the building scene are generated. This solves the problems of open vocabulary recognition and instance-level 3D mapping for unmanned tower cranes at construction sites, and realizes the restoration of real physical scale and planning interface adaptation.

CN123118935APending Publication Date: 2026-11-17CHINA CONSTR FOURTH ENG DIV CORP LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202611641826.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-22
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies struggle to achieve open-vocabulary semantic recognition, instance-level 3D semantic mapping, real-world physical scale restoration, and planning interface adaptation on construction sites, thus limiting the application of unmanned tower cranes in construction scenarios.

Method used

Images are acquired using a monocular RGB camera at the boom end of a tower crane. Combined with a multimodal large model, a cue-based segmentation model, monocular depth estimation, camera pose estimation, and tower crane slewing scale calibration, a semantic map of the building scene with real physical scale is generated. The semantic map, point cloud map, and GLB model are generated through multi-step processing.

Benefits of technology

It achieves strong open vocabulary recognition capability, high instance ID consistency, deep fusion of semantics and geometry, and real-scale control capability. The output results can directly serve the path planning and motion control of unmanned tower cranes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118935A_ABST
    Figure CN123118935A_ABST
Patent Text Reader

Abstract

This invention discloses a general semantic mapping method for tower crane scenes in architectural settings, comprising: acquiring and preprocessing original image sequences to obtain preprocessed image sequences; selecting valid frames and keyframes; inputting keyframes and / or valid frames into a multimodal large model; calculating a comprehensive matching score and assigning or inheriting global instance numbers; generating a mask and a color segmentation map based on the segmentation model; initiating a new round of segmentation tracking and merging into a global instance table with the current frame as the new initial frame; performing relative depth estimation and camera pose estimation on the preprocessed image sequences, converting the relative depth to absolute depth; first eroding the mask, then generating instance point cloud clusters, a global color point cloud map, and a point cloud segmentation map based on the eroded mask, absolute depth, and camera pose; performing outlier removal and calculating 3D bounding boxes; and outputting a semantic map, a point cloud map, and a GLB model. This invention satisfies the requirements of open-vocabulary semantic recognition, semantic mapping, and real physical scale recovery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multimodal visual perception technology, and in particular to a general scene semantic mapping method for tower cranes in architectural scenes. Background Technology

[0002] The complete operation chain of an unmanned tower crane can generally be summarized as "perception—cognition—decision—execution". The perception and cognition stages require the tower crane to obtain semantic-geometric joint information such as target categories, spatial locations, 3D dimensions, and avoidable spaces in the scene. This information is then provided to subsequent modules such as hook path planning, obstacle avoidance, lifting target positioning, and construction site digital twins in the form of semantic map files, point cloud maps, and GLB models. Therefore, open-vocabulary semantic perception, instance-level 3D reconstruction, and real-scale physical mapping in the construction scene are crucial foundations for the autonomous operation of unmanned tower cranes.

[0003] In the prior art, patent document CN121999352A discloses a water monitoring method and system based on semantic-driven and temporal behavior cognition, which focuses on open target perception, cross-frame tracking, state snapshots, and batch processing inference. However, this solution is applied to water video monitoring, and the output focuses on monitoring images and abnormal states, without addressing camera scale calibration under tower crane rotation constraints, 3D semantic point clouds of building scenes, 3D bounding boxes of objects, and map output for hook path planning. Patent document CN121259020A discloses a 3D model segmentation method, system, computing device, and electronic device, which improves the segmentation integrity of 3D models through keyframe overlay and region merging. However, this solution addresses existing 3D models and their material regions, and does not solve the problems of 3D reconstruction from real image acquisition, camera pose estimation, depth scale recovery, and tower crane decision interface at construction sites. Patent document CN121305463A discloses a method for early warning of abnormal events in large-scene surveillance videos based on a multimodal large model. This method focuses on multi-channel video preprocessing, keyframe extraction, multimodal feature fusion, and abnormal event warning. This approach leans towards security event identification and alarms, but it does not generate a metric semantic map suitable for unmanned tower crane motion planning, nor does it address the 3D localization and bounding box calculation problems for arbitrary instance targets in a building scene. While the aforementioned reference technologies can provide guidance for prompt-based identification, keyframe selection, and multi-frame merging, none of them form a complete technology chain encompassing "tower crane surround view acquisition—open semantic detection—instance-level segmentation and tracking—monocular depth and pose estimation—tower crane scale calibration—semantic point cloud fusion—semantic map and GLB output."

[0004] The existing technology has the following main shortcomings: 1. Limited semantic categories: Conventional object detection or segmentation models rely on fixed training categories, making it difficult to directly expand the recognition range through natural language when new materials, new equipment or temporary facilities appear at the construction site; 2. Semantic and geometric separation: Existing 3D reconstruction processes usually generate point clouds or meshes first, and then perform coarse-grained annotations. There is a lack of instance-level mapping from pixel-level semantic masks to 3D point clouds, which makes it difficult to stably bind object boundaries and semantic descriptions. 3. Unstable cross-frame identity: When relying solely on two-dimensional bounding boxes or segmentation masks for inter-frame association, ID jumps are prone to occur under conditions such as changes in the tower crane's panoramic view, occlusion, and stacking of similar materials. 4. Uncertainty in monocular scale: Monocular depth models such as Depth Anything typically output relative depth. If not calibrated in conjunction with the crane boom length or slewing radius, the 3D map cannot directly reflect the true distance. 5. Edge noise affects 3D localization: Segmentation boundaries and regions with abrupt changes in depth are prone to generating flying points. If the bounding box is directly calculated, it will lead to distortion in the estimation of the object's center coordinates, length, width, height, and passable space. 6. Output interface is not compatible with tower crane decision-making: Some solutions only output segmentation maps or monitoring results, lacking semantic maps, 3D bounding boxes, point cloud maps and GLB models, which are data that subsequent planning modules can directly call.

[0005] It is evident that existing technologies cannot simultaneously meet the needs of unmanned tower cranes in construction sites for open-vocabulary semantic recognition, instance-level 3D semantic mapping, real physical scale restoration, and planning interface adaptation. A general semantic mapping method that can be applied to tower cranes in construction scenarios is still needed. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a general semantic mapping method for tower crane scenes in construction sites. This method uses a monocular RGB camera at the boom end of the tower crane to collect panoramic images of the construction site. It combines a multimodal large model, a cue-based segmentation model, monocular depth estimation, camera pose estimation, tower crane rotation scale calibration, and point cloud semantic fusion to generate a semantic map of the construction scene with real physical scale.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows: This invention provides a general semantic mapping method for tower cranes in architectural scenes, comprising the following steps: Step 1: Acquire the original image sequence by using an RGB camera at the end of the tower crane boom or near the tower crane's field of view as the tower crane rotates, and then preprocess it to obtain a preprocessed image sequence; Step 2: Select valid frame preprocessed images and key frame preprocessed images from the preprocessed image sequence; Step 3: Input the preprocessed keyframe image and / or preprocessed effective frame image into the multimodal large model, and execute the processing and inference process of the multimodal large model. If necessary, perform adaptation training in advance, and obtain the output result of the multimodal large model. Step 4: Calculate the comprehensive matching score between the current candidate target and the historical target list based on the output results of the multimodal large model, assign or inherit the global instance number according to the comprehensive matching score, and re-identify and recover targets that have been lost for a short time; Step 5: Use the 2D bounding box, global instance number, and point hints as input to the segmentation model, generate a mask, and bind the global instance number with a unique color to generate a color segmentation map; Step 6: When a new global instance number appears, start a new round of segmentation and tracking with the current frame preprocessed image as the new initial frame preprocessed image, and merge the masks from multiple rounds into a global instance table according to the global instance number; Step 7: Perform relative depth estimation and camera pose estimation on the preprocessed image sequence, and convert the relative depth into absolute depth; Step 8: First, perform two-dimensional morphological erosion on the mask. Then, based on the eroded mask, absolute depth, and camera pose, back-project the two-dimensional mask pixels to the world coordinate system, aggregate them into instance point cloud clusters according to the global instance number, and generate a global color point cloud map and a point cloud segmentation map. Step 9: Remove outliers from the instance point cloud clusters and calculate the 3D bounding box based on the denoised instance point cloud clusters; Step 10: Output semantic map, point cloud map and GLB model for use by tower crane path planning, obstacle avoidance and digital twin modules.

[0008] Furthermore, step 1 specifically includes: Step 11: Use RGB cameras fixed at the front end of the tower crane boom, below the boom, or at the top of the tower crane to collect the original image sequence as the tower crane rotates. If necessary, use drones to collect supplementary images from the blind spots of the tower crane's view to add to the original image sequence. Step 12: Determine the frame extraction interval Δt based on the tower crane's slewing angular velocity ω, the camera's horizontal field of view φ_h, and the target overlap rate ρ_0, satisfying Δ_t=(1-ρ_0)φ_h / ω. If the video frame rate is f, then the frame extraction step size L is taken as an approximate integer of f×Δ_t. Based on the frame extraction step size L, obtain each original image frame from the original image sequence by extracting one frame every L. Step 13: Select a checkerboard, AprilTag, or on-site calibration board as the calibration object. After capturing the calibration object with an RGB camera to obtain a calibration image, detect the calibration feature points in the image. Use the calibration image and feature points to obtain the camera intrinsic parameters K and distortion coefficients. The camera intrinsic parameters K include the focal length fx of the RGB camera along the x-axis, the focal length fy of the RGB camera along the y-axis, the x-coordinate cx of the principal point where the camera optical axis intersects the image plane, and the y-coordinate cy of the principal point in the image pixel coordinate system. The distortion coefficients include radial distortion k1, k2, and k3, and tangential distortion p1 and p2. For a normalized pixel point (x, y) in the camera coordinate system, let r... 2 =x 2 +y 2 The distorted coordinates are: xd=x(1+k1×r 2 +k2×r 4 +k3×r 6 )+2p1xy+p2(r 2 +2x 2 ), yd=y(1+k1×r 2 +k2×r 4 +k3×r 6 )+p1(r 2 +2y2)+2p2xy, Sampling coordinates corresponding to the original image =fx×xd+cx, =fy×yd+cy, and according to =fx×xd+cx and =fy×yd+cy establishes a reverse mapping table to remap the original image to a distortion-free image; where... Represents the sampled x-coordinate in the original image. Represents the sampled ordinate in the original image; Step 14: Project the calibrated feature points back to the original image according to the camera intrinsic parameter K and distortion coefficient, and calculate the average reprojection error; if the average reprojection error exceeds the preset error threshold, re-acquire the calibration image of the calibration object selected in Step 13 and its calibration feature points and recalibrate. After the calibration is qualified, generate the distortion-free image according to Step 13; then proceed to Step 15. Specifically, the calculation of the average reprojection error is as follows: Assume each calibration image has N_cal calibration feature points, and the i-th... l The coordinates of the calibrated feature points are: The predicted projection coordinates are calculated from the known 3D coordinates of the calibrated feature points, the extrinsic parameters of the corresponding calibrated image, the camera intrinsic parameters, and the distortion coefficients. Then the average reprojection error e_rep=(1 / N_cal)×Σ[ l =1 to N_cal]sqrt(( - )^2+( - )^2); where N_cal is the total number of calibrated feature points, l For indexing feature points, The actual number detected in the calibration image l The x-coordinates of each calibrated feature point; The actual number detected in the calibration image l The ordinates of the calibrated feature points; For the first l The projected abscissa of each calibrated feature point; For the first l The projected ordinates of the calibrated feature points; sqrt represents the square root; Step 15: Normalize the brightness of the distorted image using CLAHE adaptive histogram equalization or adaptive gamma correction, and correct it according to I_norm = 255·(I / 255)^gamma, where I_norm is the corrected pixel intensity and I is the input pixel intensity of the distorted image; let the average brightness of the current frame's distorted image be I_mean and the target average brightness be I_target, then gamma = ln(I_target / 255) / ln(I_mean / 255), and limit gamma within the preset upper and lower limits; Step 16: Scale the brightness-normalized image to a uniform resolution, which is selected according to the processing requirements; if the horizontal scaling ratio is s_x and the vertical scaling ratio is s_y, then multiply fx and cx in the camera intrinsic parameters K of the distorted image by s_x, and multiply fy and cy by s_y. Step 17: The original image sequence is processed sequentially through lens distortion correction, brightness normalization, and resolution unification to obtain a preprocessed image sequence, denoted as {I1,I2,...,IN}. The timestamp, crane slewing angle, camera intrinsic parameters, and preprocessing parameters of each preprocessed image frame are recorded. The preprocessing parameters include distortion coefficients, gamma required for brightness normalization, CLAHE window and upper and lower limits, unified resolution, and corresponding horizontal and vertical scaling ratios.

[0009] Furthermore, step 2 specifically includes: Step 21: Perform sharpness check, exposure check and minimum overlap rate check on each frame of the preprocessed image sequence, and retain the images that pass all checks as valid frame preprocessed images to form a set of valid frame preprocessed images V. Step 22: Calculate the number of matching points m_t and overlap rate ρ_t=m_t / min(n_t,n_t-1) between the current frame preprocessed image and the previous keyframe preprocessed image using SuperPoint, SIFT, LoFTR or LightGlue feature matching methods, where n_t and n_t-1 are the number of scene feature points extracted and passed quality screening in the current frame preprocessed image and the previous keyframe preprocessed image, respectively. The matching point determination rules include descriptor matching and geometric verification: First, perform nearest neighbor or bidirectional nearest neighbor search on the two frames of descriptors, retain candidate points whose descriptor distance meets the preset distance threshold or nearest neighbor ratio condition, then use RANSAC to estimate the homography matrix, essential matrix or fundamental matrix, and eliminate candidate points that do not meet the geometric constraints. The finally retained candidate points are the matching points, and m_t is the number of matching points. Step 23: If ρ_t is lower than the overlap threshold, or the rotation angle difference between the current frame preprocessed image and the previous keyframe preprocessed image is greater than the angle threshold, then proceed to step 24. Step 24: Let A be the set of instances or candidate regions already covered by the current keyframe preprocessed image set K, and B_t be the set of candidate regions detected by the current frame preprocessed image. When the proportion of newly added regions Delta_t = Area(B_t-A) / Area(A∪B_t) is greater than the threshold η, the current frame preprocessed image is determined as a keyframe preprocessed image and added to the keyframe preprocessed image set K; where Area represents the number of pixels covered by the current keyframe preprocessed image set, and ∪ represents the union.

[0010] Furthermore, step 3 specifically includes: Step 31: Input the preprocessed keyframe images and / or preprocessed valid frame images into a pre-trained multimodal large model. The multimodal large model is a Qwen-VL, InternVL, GPT-4V type model or a locally deployed visual language model. Perform open vocabulary detection using construction scene cue words. The construction scene cue words require the multimodal large model to identify building materials, construction equipment, temporary facilities, personnel, safety protection devices, floor beams, columns, walls, suspended objects, and obstacles. Output a temporary target number, category name, natural language description, two-dimensional bounding box, and semantic confidence for each target. The natural language description is used to describe appearance features, including color and material features. The specific process is as follows: (1) When processing multimodal large models, the input image is divided and encoded into visual features by the visual encoder, the text encoder converts the prompts of the construction scene into text features, the cross-modal attention module establishes the association between the target region and the category words and descriptive words, the visual localization head outputs a two-dimensional bounding box, the language decoding part outputs the temporary target number, category name and natural language description, and the detection head or similarity result gives the semantic confidence. (2) If on-site proprietary component identification requires adaptation, lightweight adaptation training is performed in advance based on the pre-trained multimodal large model; if no adaptation is required, the pre-trained multimodal large model is directly used to enter the inference stage. The training data consists of tower crane on-site images, manually labeled category names, two-dimensional bounding boxes, natural language descriptions, and cross-frame IDs of the same target. The training method adopts LoRA fine-tuning or instruction fine-tuning to enable the multimodal large model to learn the correspondence between the target description, two-dimensional location, and semantic category of the construction site. (3) In the inference stage, the multimodal big model maps visual features and text features to the same semantic space and generates candidate targets through similarity calculation and visual localization head. For candidate target j, its semantic confidence s_j is: s_j=cos(E_img(region_j),E_text(prompt_j)), where region_j is the j-th candidate image region, E_img(region_j) represents the image region encoding, prompt_j is the j-th text prompt compared with the candidate image region, E_text(prompt_j) represents the text encoding, and cos represents the cosine similarity. When s_j exceeds the preset confidence threshold, the candidate target is output and a detection record set D_t is formed. Each detection record includes temporary target number, category name, natural language description, two-dimensional bounding box, semantic confidence and appearance features.

[0011] Furthermore, step 4 specifically includes: Step 41: For historical target i and current candidate target j, calculate the comprehensive matching score Score(i,j) as follows: Score(i,j)=w1×cos(e_i,e_j)+w2×IoU(B_i^pred,B_j)+w3×cos(a_i,a_j)+w4×exp(-||q_i^pred-q_j|| / σ); where i is the global instance number of the historical target, j is the candidate target number in the preprocessed image of the current frame, e_i is the semantic description embedding of historical target i, e_j is the semantic description embedding of current candidate target j, and IoU represents the intersection area and union area of ​​the two two-dimensional bounding boxes. The ratio of the area of ​​the set, B_i^pred is the 2D bounding box predicted by the camera pose and historical trajectory of the current frame preprocessed image, B_j is the currently detected 2D bounding box, a_i is the appearance feature of the historical target i, a_j is the appearance feature of the current candidate target j, q_i^pred is the position of the historical target i in the image or preliminary 3D space, q_j is the position of the current candidate target j in the image or preliminary 3D space, ||q_i^pred-q_j|| represents the Euclidean distance between q_i^pred and q_j, σ is the spatial distance normalization parameter; w1 to w4 are the corresponding weights and w1+w2+w3+w4=1; Step 42: During matching, construct a score matrix and use the Hungarian algorithm or a greedy strategy to obtain a one-to-one correspondence. When Score(i,j) is greater than the threshold θ_match, the current candidate target j inherits the obj_id of the historical target i. obj_id represents the unique global instance number within the scanning cycle. When the comprehensive matching score of all candidate targets is less than the threshold θ_match, it is determined as a new target and a new obj_id is assigned. Step 43: If a historical target does not appear in the current frame of the preprocessed image, it enters a short-term loss state and records missing_count. missing_count is the number of consecutive frames in which the historical target has not been matched in the current frame of the preprocessed image. When missing_count does not exceed M frames of preprocessed images and subsequent frames of preprocessed images show candidate targets with semantic and spatial matching scores exceeding the re-identification threshold θ_reid, the original obj_id is restored. When missing_count exceeds M or the candidate target leaves the known field of view, the trajectory of the candidate target is terminated. Step 44: Output the detection set G_t with obj_id. Each obj_id is unique throughout the entire scanning period and is bound to the natural language semantic description, historical occurrence frames, and two-dimensional bounding box sequence.

[0012] Furthermore, step 5 specifically includes: Step 51: Take the preprocessed image I_t as the image input, and take the two-dimensional bounding box, obj_id, and point cue generated by the center of the two-dimensional bounding box or the historical mask as the cue input to the segmentation model; Step 52: The image encoder of the segmentation model extracts the image embedding from the preprocessed image I_t. The cue encoder encodes the two-dimensional bounding box or point cue into a cue embedding. The mask decoder fuses the image embedding and the cue embedding and outputs the mask Mask_t(i) and the mask confidence, where i is the global instance number of the historical target. If the mask resolution output by the segmentation model is different from that of the preprocessed image, it is mapped back to the preprocessed image using u=scale_x×u_low, v=scale_y×v_low, where u is the x-coordinate mapped to the preprocessed image, v is the y-coordinate mapped to the preprocessed image; scale_x is the horizontal scaling ratio, scale_y is the vertical scaling ratio; u_low is the x-coordinate in the low-resolution mask, v_low is the y-coordinate in the low-resolution mask; scale_x=W / W_low, scale_y=H / H_low, W is the width of the preprocessed image, H is the height of the preprocessed image, W_low is the width of the low-resolution mask, and H_low is the height of the low-resolution mask. Step 53: Bind a unique color to each obj_id. The color is generated by looking up a table or a hash function. Establish an obj_id-color mapping table and initialize a blank image of the same size as the preprocessed image. Write the pixels in Mask_t(i) with the color corresponding to the obj_id to generate a color segmentation image S_t. Step 54: Establish the correspondence between the mask and obj_id, and assign the pixels whose Euclidean distance between the pixel color in the color segmentation image S_t and the color in the obj_id-color mapping table is less than the threshold τ_color to the corresponding obj_id.

[0013] Furthermore, step 6 specifically includes: Step 61: When it is determined that there is a new obj_id in the current frame preprocessing image, the current frame preprocessing image is used as the new initial frame preprocessing image of the target, and a new round of SAM3 segmentation tracking is started using its two-dimensional bounding box. The mask sequence obtained in this round of tracking is added to the global segmentation result. Step 62: During merging, use obj_id as the primary key to unify the masks, mask confidence and time ranges from different rounds into the global instance table. The time range is determined by the frame number of the first valid mask generated for the same obj_id and the frame number of the last valid mask generated. Step 63: If the same pixel is covered by multiple target masks, retain the target with the highest P among the targets covering the pixel according to priority P = β1×c_mask+β2×c_track+β3×c_sem; if P is the same, retain the target with the smallest obj_id, where c_mask is the mask confidence, c_track is the normalized comprehensive matching score, c_sem is the cosine similarity of the semantic description embedding, and β1, β2 and β3 are the corresponding weights and β1+β2+β3=1; Step 64: If the IoU of two masks exceeds the threshold θ_merge and the semantic description similarity exceeds the threshold θ_sem, then they are considered as the same target and their trajectories are merged; if the IoU of two masks does not exceed the threshold θ_merge or the semantic description similarity does not exceed the threshold θ_sem, then they are kept as different instances, and the color segmentation map S_t of each frame, the mask sequence of each instance, and the global instance table are output.

[0014] Furthermore, step 7 specifically includes: Step 71: Input the preprocessed image I_t into the monocular depth estimation model, and output the relative depth D_rel_t(u,v) after passing through the DINOv2 / ViT encoder and DPT decoder. If necessary, distillation or fine-tuning can be performed in advance using field data with depth or SfM sparse depth. Otherwise, the pre-trained monocular depth estimation model can be used directly for inference. Step 72: For adjacent preprocessed images I_t and I_t+1, extract feature points, estimate the essential matrix E by combining the camera intrinsic parameters K, and obtain the relative rotation and translation directions by E decomposition. Then, optimize the camera pose C_t=(R_t,t_t) for each frame through PnP, triangulation and bundle adjustment, where R_t represents the rotation component from the camera coordinate system to the world coordinate system, and t_t represents the relative translation component from the camera coordinate system to the world coordinate system. The world coordinate system is selected as the origin near the camera pose of the first frame or the center of rotation of the tower crane. Step 73: Extract the camera optical center O_t from the camera pose of each frame, and obtain the estimated rotation center O_c and estimated radius R_est=(1 / N)·Σ_t||O_t-O_c|| through least squares circle fitting, where N is the number of effective camera optical centers that participate in circle fitting and are removed by outlier elimination. The actual horizontal distance R_real from the tower crane camera to the rotation center is measured on site, and then the tower crane scale factor α=R_real / R_est; Step 74: Linearly align the relative depth D_rel_t(u,v) with the relative depth obtained from SfM sparse points or pose triangulation to obtain D_align_t(u,v) = a_t × D_rel_t(u,v) + b_t, where D_align_t(u,v) is the depth aligned to the SfM reconstruction scale, and a_t and b_t are obtained by least-squares fitting of sparse depth points in the same preprocessed image frame. Then, use the crane scale factor to restore the absolute depth D_abs_t(u,v): D_abs_t(u,v) = α × D_align_t(u,v); At the same time, with the origin of the world coordinate system as the scaling center, convert the relative translation component t_t to the absolute translation component t_abs_t: t_abs_t = α × t_t, while keeping the rotation component R_t unchanged, to obtain the metric pose C_abs_t = (R_t, t_abs_t); Step 75: Calculate the depth gradient g(u,v) = sqrt(( D_abs_t(u,v) / u)^2+( D_abs_t(u,v) / v)^2), where sqrt is the square root operation. D_abs_t(u,v) / u represents the rate of change of depth along the horizontal direction of the image. D_abs_t(u,v) / v represents the rate of change of depth along the longitudinal direction of the image. When g(u,v) > threshold τ_depth, the pixel is considered to be in a depth discontinuity region, which together with the segmentation mask boundary forms an edge risk mask.

[0015] Furthermore, step 8 specifically includes: Step 81: For the mask Mask_t(i) of obj_id=i in frame t, first perform two-dimensional morphological erosion to obtain Mask_eroded, then take the pixel (u,v) and its absolute depth D_abs_t(u,v) within Mask_eroded, and back-project it to the camera coordinate system in combination with the camera intrinsic parameter K: P_cam=D_abs_t(u,v)×K^(-1)×[u,v,1]^T, where P_cam represents the coordinates of the three-dimensional point corresponding to the pixel in the current camera coordinate system; where u is the abscissa after mapping to the preprocessed image, and v is the ordinate after mapping to the preprocessed image; Specifically, the two-dimensional morphological erosion is: Mask_eroded = Mask_t(i) E, where E is the structuring element, and Mask_eroded represents the mask obtained by eroding Mask_t(i) through the structuring element E; the structuring element is a 3×3 or 5×5 convolution kernel. This indicates an erosion operation, where the size of the erosion core is adaptively selected based on the target area. Step 82: Transform P_cam to the unified world coordinate system using the metric pose C_abs_t=(R_t,t_abs_t): P_world=R_t×P_cam+t_abs_t, where P_world is the world coordinate point corresponding to the pixel; Step 83: Accumulate the P_world values ​​of the same obj_id across all frames to form an instance point cloud cluster PC_i; accumulate all RGB-D points that are not filtered by instance to form a global color point cloud map PC_scene; accumulate all points with obj_id color and semantic labels to form a point cloud segmentation map PC_semantic; where RGB-D indicates that the same pixel has both RGB color value and absolute depth value.

[0016] Furthermore, step 9 specifically includes: Step 91: For each point p_k in the instance point cloud cluster PC_i, calculate the average distance d_k between it and its k nearest neighbors. Then calculate the mean μ and standard deviation σ_d of all d_k. If d_k > μ + λ × σ_d, then p_k is determined to be a statistical outlier and removed. Here, λ is the standard deviation multiple of the outlier determination threshold, which is used to control the strictness of noise reduction and is taken as 1.0 to 2.0. Step 92: Calculate the 3D bounding box for the denoised PC_i. If the axisymmetric bounding box AABB is used, calculate x_min, x_max, y_min, y_max, z_min, and z_max respectively, with the center coordinates as ((x_min+x_max) / 2, (y_min+y_max) / 2, (z_min+z_max) / 2) and the dimensions as (x_max-x_min, y_max-y_min, z_max-z_min). Here, x_min is the minimum value of the point cloud in the x-axis direction, x_max is the maximum value of the point cloud in the x-axis direction, y_min is the minimum value of the point cloud in the y-axis direction, y_max is the maximum value of the point cloud in the y-axis direction, z_min is the minimum value of the point cloud in the z-axis direction, and z_max is the maximum value of the point cloud in the z-axis direction. If the target is placed obliquely and AABB generates an empty envelope, then the PCA principal axis direction is used, and the extreme values ​​of each axis in the principal axis coordinate system are taken to form the OBB. Step 10 specifically includes: Step 101: Output three types of results: semantic map, point cloud map, and GLB model, as detailed below: (1) The semantic map is stored in JSON, GeoJSON or database table format. Each object record contains at least obj_id, category name, natural language semantic description, first and last appearance frames, three-dimensional center coordinates, three-dimensional size, bounding box type, number of point clouds, confidence level, color encoding and index relationship with the original image frame; (2) The point cloud map includes a global color point cloud map and a point cloud segmentation map with obj_id color and semantic labels; (3) The GLB model is generated by point cloud through TSDF fusion, Poisson surface reconstruction or system meshing, and at least the scene nodes and their transformations, mesh vertices, triangular face indices, normal vectors, materials or colors, unit scale and world coordinate system are recorded. When it is necessary to retain instance semantics, the obj_id, category name and its association with the semantic map record are recorded in the node name or additional fields. Step 102: Provide the semantic map, point cloud map, and GLB model to the tower crane decision module, planning module, and execution module. The decision module reads the semantic map to obtain the 3D bounding boxes of the target object and obstacles. The planning module calculates the travel path of the hook from the starting point to the target point in the GLB or voxel map. The execution module converts the planned path into tower crane slewing, luffing, and hoisting control commands. If the scan finds that an obstacle has been added or its position has changed, the path verification is retried by incrementally updating the semantic map.

[0017] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: 1. Strong open vocabulary recognition capability: It can recognize any target on the construction site through multimodal large models and prompt words, and is not limited to fixed training categories.

[0018] 2. Consistent Instance ID (obj_id) Time Sequence: By combining semantic, appearance, geometry and camera motion matching, the problems of occlusion, short-term missed detection and the appearance of new targets in the tower crane surround view process are solved.

[0019] 3. Semantic and geometric fusion: The two-dimensional instance mask, depth map and pose are uniformly mapped into a three-dimensional semantic point cloud, directly forming a joint representation of "object-position-size".

[0020] 4. Real-world dimensions can be used for control: The slewing radius of the tower crane is used for dimensional calibration, so that the monocular reconstruction results have physical dimensions, which can meet the needs of path planning and motion control.

[0021] 5. More stable positioning accuracy: The combination of 2D mask erosion and 3D statistical outlier removal reduces edge noise and improves the reliability of 3D bounding boxes.

[0022] 6. Output can directly serve unmanned tower cranes: Semantic maps, point cloud maps, and GLB models can be called by path planning, obstacle avoidance, digital twins, and construction management systems. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is an execution flowchart of a general scene semantic mapping method for tower cranes in architectural scenes provided by an embodiment of the present invention.

[0025] Figure 2 This is a flowchart of cross-frame ID matching provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of tower crane dimension calibration provided in an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the application of semantic map and GLB model service path planning provided in the embodiments of the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figures 1-4 The present invention provides a general semantic mapping method for tower cranes in architectural scenes, comprising the following steps: Step S1, Tower Crane Surround View Image Acquisition and Preprocessing: The original image sequence is acquired by an RGB camera at the end of the tower crane boom or near the tower crane's field of view as the tower crane rotates, and then preprocessed to obtain a preprocessed image sequence; In this embodiment, step S1 specifically includes: Step S11: Use RGB cameras fixed at the front end of the tower crane boom, below the boom, or at the top of the tower crane to collect the original image sequence (image sequence of the construction site) as the tower crane rotates. If necessary, use drones to collect supplementary images from the blind spots of the tower crane's view to add to the original image sequence. Step S12: Determine the frame extraction interval Δ_t based on the tower crane's rotational angular velocity ω, the camera's horizontal field of view φ_h, and the target overlap rate ρ_0, satisfying Δ_t = (1-ρ_0)φ_h / ω. The effective overlap area of ​​adjacent frames is preferably maintained between 40% and 60%, with a minimum of 30%. If the overlap rate is higher than 60%, adjacent frame information is repeated, increasing the computational load for detection, segmentation, and point cloud fusion. If it is lower than 30%, there are insufficient stable matching points, easily causing pose estimation failure, obj_id jumps, and point cloud breaks. For example, when the tower crane rotates at 1° / s and extracts one frame every 10°, approximately 36 frames of panoramic images can be obtained in a single 360° scan. If the video frame rate is f, the frame extraction step size L is an approximate integer of f × Δ_t. Based on the frame extraction step size L, each original image frame is obtained by extracting one frame every L from the original image sequence. The original image is then preprocessed, including lens distortion correction, brightness normalization, resolution unification, and timestamp recording. Step S13: Select a checkerboard, AprilTag, or on-site calibration board as the calibration object. After capturing the calibration object with an RGB camera to obtain a calibration image, detect the calibration feature points in the calibration image. Use the calibration image and calibration feature points to obtain the camera intrinsic parameters K and distortion coefficients. The camera intrinsic parameters K include the focal length fx of the RGB camera in the x-axis direction, the focal length fy of the RGB camera in the y-axis direction, the x-coordinate cx and y-coordinate cy of the principal point where the camera optical axis intersects the image plane in the image pixel coordinate system. The distortion coefficients include radial distortion k1, k2, and k3, and tangential distortion p1 and p2. For a normalized pixel point (x, y) in the camera coordinate system, let r... 2 =x 2 +y 2 The distorted coordinates are: xd=x(1+k1×r 2 +k2×r 4 +k3×r 6 )+2p1xy+p2(r 2 +2x 2 ), yd=y(1+k1×r 2 +k2×r 4 +k3×r 6 )+p1(r 2 +2y2)+2p2xy, Sampling coordinates corresponding to the original image =fx×xd+cx, =fy×yd+cy, and according to =fx×xd+cx and =fy×yd+cy A reverse mapping table is established to remap the original image to a distortion-free image. When establishing the reverse mapping table, each target pixel in the distortion-free image is traversed. First, normalized coordinates are obtained using K^-1, then its sampling position in the original image is calculated according to the distortion model and recorded. During remapping, this position is interpolated to obtain the distortion-free image, thereby reducing feature matching and pose estimation errors; where r 2 k1×r represents the squared radial distance from the normalized point to the principal point. 2 +k2×r 4 +k3×r 6 This is the radial distortion correction amount; Represents the sampled x-coordinate in the original image. Represents the sampled ordinate in the original image; Step S14: Project the calibrated feature points back to the original image according to the camera intrinsic parameter K and distortion coefficient, and calculate the average reprojection error; if the average reprojection error exceeds the preset error threshold, re-acquire the calibration image of the calibration object selected in step S13 and its calibration feature points and recalibrate. After the calibration is qualified, generate the distortion-free image according to step S13; then proceed to step S15. Specifically, the calculation of the average reprojection error is as follows: Assume each calibration image has N_cal calibration feature points, and the i-th... l The coordinates of the calibrated feature points are: The predicted projection coordinates are calculated from the known 3D coordinates of the calibrated feature points, the extrinsic parameters of the corresponding calibrated image, the camera intrinsic parameters, and the distortion coefficients. Then the average reprojection error e_rep=(1 / N_cal)×Σ[ l =1 to N_cal]sqrt(( - )^2+( - )^2); where N_cal is the total number of calibrated feature points, l For indexing feature points, The actual number detected in the calibration image l The x-coordinates of each calibrated feature point; The actual number detected in the calibration image l The ordinates of the calibrated feature points; For the first l The projected abscissa of each calibrated feature point; For the first l The projected ordinates of the calibrated feature points; sqrt represents the square root; Step S15: Normalize the brightness of the distorted image using CLAHE (Contrast Limited Adaptive Histogram Equalization) or adaptive gamma correction, with correction based on I_norm = 255·(I / 255)^gamma, where I_norm is the corrected pixel intensity and I is the input pixel intensity of the distorted image. Let the average brightness of the current frame's distorted image be I_mean and the target average brightness be I_target, then gamma = ln(I_target / 255) / ln(I_mean / 255), and limit gamma within a preset upper and lower limit range. When I_mean is lower than the target brightness, gamma is less than 1 to brighten; when I_mean is higher than the target brightness, gamma is greater than 1 to darken. Step S16: Scale the brightness-normalized image to a uniform resolution, which is selected according to processing requirements, such as 1920×1080 or 1280×720; wherein, the camera intrinsic parameter K used in subsequent steps must be consistent with the distortion-reduced and scaled image; if the horizontal scaling ratio is s_x and the vertical scaling ratio is s_y, then multiply fx and cx in the camera intrinsic parameter K of the distortion-reduced image by s_x, and multiply fy and cy by s_y; Step S17: The original image sequence is sequentially processed through lens distortion correction, brightness normalization, and unified resolution to obtain a preprocessed image sequence, denoted as {I1,I2,...,IN}. The timestamp, crane slewing angle, camera intrinsic parameters, and preprocessing parameters for each preprocessed image frame are recorded. The preprocessing parameters include distortion coefficients, gamma required for brightness normalization, the CLAHE window and its upper and lower limits, unified resolution, and the corresponding horizontal and vertical scaling ratios. The timestamp and crane slewing angle are used to determine the frame order and verify the frame extraction angle interval, while the preprocessing parameters are used to reproduce image correction. The current frame, adjacent frames, keyframes, valid frames, and frame indices in steps S2 to S10 all correspond to this preprocessed image sequence; the frames before preprocessing in step S1 belong to the original image sequence.

[0029] Step S2, Keyframe and Valid Frame Selection: Select valid frame preprocessed images and keyframe preprocessed images from the preprocessed image sequence; In order to reduce redundant calculations and improve multi-view coverage, the present invention selects keyframes from the preprocessed image sequence, while retaining all valid frames for subsequent segmentation tracking and point cloud fusion.

[0030] In this embodiment, step S2 specifically includes: Step S21: Perform sharpness check, exposure check and minimum overlap rate check on each frame of the preprocessed image sequence, and retain the images that pass all checks as valid frame preprocessed images to form a set V of valid frame preprocessed images. Step S22: Calculate the number of matching points m_t and overlap rate ρ_t = m_t / min(n_t, n_t-1) between the current frame preprocessed image and the previous keyframe preprocessed image using SuperPoint, SIFT, LoFTR, or LightGlue feature matching methods. Here, n_t and n_t-1 are the number of scene feature points extracted and passed quality screening in the current frame preprocessed image and the previous keyframe preprocessed image, respectively. SuperPoint: A deep learning-based feature point extraction and descriptor network. It can simultaneously detect key points in an image and generate descriptors, exhibiting strong robustness to changes in illumination and viewing angle.

[0031] SIFT stands for Scale-Invariant Feature Transform. It is a classic local feature extraction algorithm in the field of computer vision, which is invariant to changes in image scale, rotation, and brightness.

[0032] LoFTR stands for Detector-Free Local Feature Matching with Transformers. It eliminates the need for prior keypoint extraction, instead leveraging the Transformer architecture to establish dense matching between images at both coarse and fine granular levels, making it particularly suitable for low-texture construction site scenes.

[0033] LightGlue: A lightweight feature matching network (an improved version of SuperGlue). It is used for fast and accurate pairing of extracted feature points with high computational efficiency.

[0034] The matching point determination rules include descriptor matching and geometric verification: First, nearest neighbor or bidirectional nearest neighbor search is performed on the descriptors of two frames to retain candidate points whose descriptor distance meets a preset distance threshold or nearest neighbor ratio condition. Then, RANSAC is used to estimate the homography matrix, essential matrix, or fundamental matrix, and candidate points that do not meet the geometric constraints are eliminated. The final retained candidate points are the matching points, where m_t is the number of matching points. RANSAC stands for Random Sample Consensus. It is an iterative algorithm used to estimate the parameters of a mathematical model (such as the homography matrix, essential matrix, or fundamental matrix) from a dataset containing a large number of mismatches (outside points). It is a standard method for eliminating mismatches and retaining geometric interior points.

[0035] Step S23: If ρ_t is lower than the overlap threshold, or the rotation angle difference between the current frame preprocessed image and the previous key frame preprocessed image is greater than the angle threshold, then proceed to step S24. Step S24: Let A be the set of instances or candidate regions already covered by the current keyframe preprocessed image set K, and B_t be the set of candidate regions detected by the current frame preprocessed image. When the proportion of newly added regions Delta_t = Area(B_t-A) / Area(A∪B_t) is greater than the threshold η, the current frame preprocessed image is determined as a keyframe preprocessed image and added to the keyframe preprocessed image set K; where Area represents the number of pixels covered by the current keyframe preprocessed image set, and ∪ represents the union.

[0036] Step S3, Multimodal Large Model Open Semantic Detection: Input the preprocessed keyframe image and / or the preprocessed effective frame image into the multimodal large model, and perform the processing and inference process of the multimodal large model. If necessary, perform pre-adaptation training and obtain the output result of the multimodal large model. In this embodiment, step S3 specifically includes: Step S31: Input the preprocessed keyframe images and / or preprocessed valid frame images into a pre-trained multimodal large model. The multimodal large model can be a Qwen-VL (Tongyi Qianwen Visual Language Large Model, supporting multimodal tasks such as image understanding and text generation), InternVL (Shusheng·Puyu Visual Language Large Model, possessing powerful multimodal perception and dialogue capabilities), GPT-4V (GPT-4 Vision, a multimodal large model with visual understanding capabilities, used for open-vocabulary object detection and scene understanding) type model or a locally deployed visual language model. Perform open-vocabulary detection using construction scene prompts. These prompts require the multimodal large model to identify building materials, construction equipment, temporary facilities, personnel, safety protection devices, floor beams, columns, walls, suspended objects, and obstacles. For each target, output a temporary target number, category name, natural language description, two-dimensional bounding box, and semantic confidence. The natural language description describes appearance features, including color and material features. The specific process is as follows: (1) When processing multimodal large models, the input image is divided and encoded into visual features by the visual encoder, the text encoder converts the prompts of the construction scene into text features, the cross-modal attention module establishes the association between the target region and the category words and descriptive words, the visual localization head outputs a two-dimensional bounding box, the language decoding part outputs the temporary target number, category name and natural language description, and the detection head or similarity result gives the semantic confidence. (2) If on-site proprietary component identification requires adaptation, lightweight adaptation training is performed in advance based on the pre-trained multimodal large model; if no adaptation is required, the pre-trained multimodal large model is directly used to enter the inference stage. The training data consists of tower crane on-site images, manually labeled category names, two-dimensional bounding boxes, natural language descriptions, and cross-frame IDs of the same target. The training method adopts LoRA fine-tuning or instruction fine-tuning to enable the multimodal large model to learn the correspondence between the target description, two-dimensional location, and semantic category of the construction site. (3) In the inference stage, the multimodal big model maps visual features and text features to the same semantic space and generates candidate targets through similarity calculation and visual localization head. For candidate target j, its semantic confidence s_j is: s_j=cos(E_img(region_j),E_text(prompt_j)), where region_j is the j-th candidate image region, E_img(region_j) represents the image region encoding, prompt_j is the j-th text prompt compared with the candidate image region, E_text(prompt_j) represents the text encoding, and cos represents the cosine similarity. When s_j exceeds the preset confidence threshold, the candidate target is output and a detection record set D_t is formed. Each detection record includes temporary target number, category name, natural language description, two-dimensional bounding box, semantic confidence and appearance features.

[0037] Step S4, Cross-frame semantic matching and global ID allocation: Calculate the comprehensive matching score between the current candidate target and the historical target list based on the output of the multimodal large model, allocate or inherit the global instance number according to the comprehensive matching score, and re-identify and recover targets lost in a short period of time; In this embodiment, step S4 specifically includes: Step S41: For historical target i and current candidate target j, calculate the comprehensive matching score Score(i,j) as follows: Score(i,j)=w1×cos(e_i,e_j)+w2×IoU(B_i^pred,B_j)+w3×cos(a_i,a_j)+w4×exp(-||q_i^pred-q_j|| / σ); where i is the global instance number of the historical target, j is the candidate target number in the preprocessed image of the current frame, e_i is the semantic description embedding of historical target i, e_j is the semantic description embedding of current candidate target j, IoU represents the ratio of the intersection area to the union area of ​​two two-dimensional bounding boxes, the larger the value, the higher the overlap of the boxes, B_i^pred, and IoU represents the ratio of the intersection area to the union area of ​​two two-dimensional bounding boxes. `red` represents the 2D bounding box predicted from the camera pose and historical trajectories of the preprocessed image of the current frame; `B_j` represents the currently detected 2D bounding box; `a_i` represents the appearance features of historical target `i`; `a_j` represents the appearance features of the current candidate target `j`; `q_i^pred` represents the position of historical target `i` in the image or preliminary 3D space; `q_j` represents the position of the current candidate target `j` in the image or preliminary 3D space; `||q_i^pred-q_j||` represents the Euclidean distance between `q_i^pred` and `q_j`. The exponent term uses a negative sign to decrease the term as the distance increases. Therefore, the closer the positions are, the higher the spatial matching contribution. `σ` is the spatial distance normalization parameter; `w1` to `w4` are the corresponding weights, and `w1+w2+w3+w4=1`. Step S42: During matching, a score matrix is ​​constructed and a one-to-one correspondence is obtained using the Hungarian algorithm or a greedy strategy. When Score(i,j) is greater than the threshold θ_match, the current candidate target j inherits the obj_id of the historical target i. obj_id represents the unique global instance number within the scanning cycle. When the comprehensive matching score of all candidate targets is less than the threshold θ_match, it is determined as a new target and a new obj_id is assigned. In this algorithm, each row of the score matrix corresponds to a historical target, and each column corresponds to a current candidate target. The matrix elements are Score(i,j). The Hungarian algorithm selects non-conflicting row and column combinations with the goal of maximizing the total matching score. When using a greedy strategy, it selects unused target pairs in descending order of score. The target pairs obtained by the algorithm still need to pass the θ_match threshold; those that fail are treated as unmatched.

[0038] Step S43: If a historical target does not appear in the current frame of the preprocessed image, it enters a short-term loss state and records missing_count. missing_count is the number of consecutive frames in which the historical target has not been matched in the current frame of the preprocessed image. When missing_count does not exceed M frames of preprocessed images and subsequent frames of preprocessed images show candidate targets with semantic and spatial matching scores exceeding the re-identification threshold θ_reid, the original obj_id is restored. When missing_count exceeds M or the candidate target leaves the known field of view, the trajectory of the candidate target is terminated. This continuity judgment is used to handle occlusion, tower crane perspective switching, and short-term missed detection.

[0039] Step S44: Output a detection set G_t with obj_id, where each obj_id is unique throughout the entire scanning period, and is associated with natural language semantic description, historical occurrence frames, and two-dimensional bounding box sequence.

[0040] Step S5, SAM3 cue-based instance segmentation and color coding: The two-dimensional bounding box, global instance number, and point cue are used as cue inputs to the segmentation model to generate a mask, and the global instance number is bound to a unique color to generate a color segmentation map; In this embodiment, step S5 specifically includes: Step S51: Take the preprocessed image I_t as the image input, and take the two-dimensional bounding box, obj_id, and point cue generated by the center of the two-dimensional bounding box or the historical mask as the cue input to the segmentation model (such as SAM3 or a similar model that supports cueing). Step S52: The image encoder of the segmentation model extracts the image embedding from the preprocessed image I_t. The cue encoder encodes the two-dimensional bounding box or point cue into a cue embedding. The mask decoder fuses the image embedding and the cue embedding and outputs the mask Mask_t(i) and the mask confidence, where i is the global instance number of the historical target. If the mask resolution output by the segmentation model is different from that of the preprocessed image, it is mapped back to the preprocessed image through u=scale_x×u_low, v=scale_y×v_low, where u is the horizontal coordinate after mapping to the preprocessed image, v is the vertical coordinate after mapping to the preprocessed image; scale_x is the horizontal scaling ratio, scale_y is the vertical scaling ratio; u_low is the horizontal coordinate in the low-resolution mask, v_low is the vertical coordinate in the low-resolution mask; scale_x=W / W_low, scale_y=H / H_low, W is the width of the preprocessed image, H is the height of the preprocessed image, W_low is the width of the low-resolution mask, and H_low is the height of the low-resolution mask. Step S53: Bind a unique color to each obj_id. The color is generated by looking up a table or a hash function. For example, Color(obj_id) = Hash(obj_id) mod 256 to get the RGB three-channel value, where Hash is the hash function and mod is the modulo operator. Establish an obj_id-color mapping table and initialize a blank image of the same size as the preprocessed image. Write the pixels in Mask_t(i) with the color corresponding to the obj_id to generate a color segmentation image S_t. Step S54: Establish the correspondence between the mask and obj_id for use in subsequent step S8. Both 3D backprojection and writing to the instance point cloud cluster PC_i by obj_id are performed in step S8. Pixels whose Euclidean distance between the pixel color in the color segmentation map S_t and the color in the obj_id-color mapping table is less than the threshold τ_color are assigned to their corresponding obj_id to avoid color errors caused by compression, anti-aliasing, or rendering interpolation. This color recognition and filtering rule ensures consistency for the same instance in 2D segmentation map, semantic point cloud, and 3D bounding box calculations.

[0041] Step S6, Multi-round new target segmentation and result merging: When a new global instance number appears, start a new round of segmentation and tracking with the current frame preprocessed image as the new initial frame preprocessed image, and merge the multi-round masks into a global instance table according to the global instance number; In this embodiment, step S6 specifically includes: Step S61: Adopt a multi-round new target segmentation strategy: When it is determined that there is a new obj_id in the current frame preprocessed image, the current frame preprocessed image is used as the new initial frame preprocessed image of the target, and a new round of SAM3 segmentation tracking is started using its two-dimensional bounding box, and the mask sequence obtained in this round of tracking is added to the global segmentation result. Step S62: During merging, using obj_id as the primary key, the masks, mask confidence and time ranges from different rounds are unified into the global instance table. The time range is determined by the frame number of the first valid mask generated for the same obj_id and the frame number of the last valid mask generated. Step S63: If the same pixel is covered by multiple target masks, retain the target with the highest P among the targets covering the pixel according to priority P=β1×c_mask+β2×c_track+β3×c_sem; if P is the same, retain the target with the smallest obj_id, where c_mask is the mask confidence, c_track is the normalized comprehensive matching score, c_sem is the cosine similarity of the semantic description embedding, and β1, β2 and β3 are the corresponding weights and β1+β2+β3=1; Step S64: If the IoU of two masks exceeds the threshold θ_merge and the semantic description similarity exceeds the threshold θ_sem, then they are considered as the same target and their trajectories are merged; if the IoU of two masks does not exceed the threshold θ_merge or the semantic description similarity does not exceed the threshold θ_sem, then they are retained as different instances, and the color segmentation map S_t of each frame, the mask sequence of each instance, and the global instance table are output.

[0042] Step S7, Depth Estimation, Camera Pose Estimation and Tower Crane Dimension Calibration: Perform relative depth estimation and camera pose estimation on the preprocessed image sequence, and convert the relative depth into absolute depth; In this embodiment, step S7 specifically includes: Step S71: Input each preprocessed image I_t into a monocular depth estimation model (such as Depth Anything 3 or a similar model). After passing through the DINOv2 / ViT (DINOv2: a self-supervised visual Transformer model proposed by Meta, often used as a visual encoder to extract high-quality image features. ViT: full name of Vision Transformer) encoder and DPT (Dense Prediction Transformer) decoder, the relative depth D_rel_t(u,v) is output. If necessary (e.g., in construction scenes with dust, strong reflections, or nighttime conditions), distillation or fine-tuning is performed in advance using field data with depth or SfM (Structure from Motion) sparse depth. Otherwise, the pre-trained monocular depth estimation model is used directly for inference to make the depth boundary more suitable for building materials and structures. Step S72: For adjacent preprocessed images I_t and I_t+1, extract feature points, estimate the essential matrix E by combining the camera intrinsic parameters K, and obtain the relative rotation and translation directions by E decomposition. Then, optimize the camera pose C_t=(R_t,t_t) for each frame through PnP, triangulation and bundle adjustment, where R_t represents the rotation component from the camera coordinate system to the world coordinate system, and t_t represents the relative translation component from the camera coordinate system to the world coordinate system. The world coordinate system is selected as the origin near the camera pose of the first frame or the center of rotation of the tower crane. During calculation, the matching pixels p and p' are first converted to normalized coordinates using x = K^-1 × p and x' = K^-1 × p'. Then, the epipolar constraint x'^T × E × x = 0 is used to calculate the essential matrix E using a five-point or eight-point method combined with RANSAC. Since E = [t_rel]_x × R_rel, E can be decomposed, and the correct relative rotation R_rel and translation direction t_rel of adjacent frames can be selected by ensuring that the triangulation point is in front of the camera. For the triangulated 3D point X_k, PnP calculates the extrinsic parameters R_cw and t_cw from the world to the camera by minimizing the reprojection error sum_k||p_k-pi(K×(R_cw×X_k+t_cw))||^2. Then, it converts the pose from the camera to the world by taking R_t=R_cw^T and t_t=-R_cw^T×t_cw. Here, pi represents perspective division, and X_k and t_cw use the same reconstruction scale. Finally, bundle adjustment is used to jointly adjust the pose and 3D points of each frame to minimize the total reprojection error of all observations.

[0043] Step S73: To solve the monocular scale ambiguity, this invention utilizes the kinematics of tower crane rotation for scale calibration. The camera optical center O_t is extracted from the camera pose of each frame. The estimated rotation center O_c and estimated radius R_est = (1 / N)·Σ_t||O_t-O_c|| are obtained through least-squares circle fitting, where N is the number of effective camera optical centers participating in circle fitting and removed through outlier elimination. The actual horizontal distance R_real from the tower crane camera to the rotation center is measured on-site, for example, by measuring the tower crane boom length, camera installation position, and rotation center position. Then, the tower crane scale factor α = R_real / R_est; Step S74: Linearly align the relative depth D_rel_t(u,v) with the relative depth obtained from SfM sparse points or pose triangulation to obtain D_align_t(u,v) = a_t × D_rel_t(u,v) + b_t, where D_align_t(u,v) is the depth aligned to the SfM reconstruction scale, and a_t and b_t are obtained by least-squares fitting of sparse depth points in the same preprocessed image frame. Then, use the crane scale factor to recover the absolute depth D_abs_t(u,v): D_abs_t(u,v) = α × D_align_t(u,v) v); Simultaneously, using the origin of the world coordinate system as the scaling center, the relative translation component t_t is converted into the absolute translation component t_abs_t: t_abs_t = α × t_t, while the rotation component R_t remains unchanged, resulting in the metric pose C_abs_t = (R_t, t_abs_t), which is used in step S8; if there are insufficient sparse depth points, the normalized relative depth can be directly multiplied by α as the approximate depth to be verified, or the depth of this frame can be discarded; the previous branch is not considered as having completed scale calibration, and must undergo multi-frame geometric consistency verification and scale alignment of other calibrated frames before it can participate in the metric point cloud fusion. Here, the origin and coordinate axis direction of the world coordinate system described in step S72 remain unchanged, and only the length scale is converted to the actual physical scale.

[0044] Let the relative depth of the k-th sparse point in the same frame be x_k = D_rel_t(u_k, v_k), and the corresponding depth given by triangulation or SfM be y_k. Then a_t and b_t are obtained by minimizing sum_k(a_t × x_k + b_t - y_k)^2. The closed-form result can be written as a_t = sum_k[(x_k - x_mean)(y_k - y_mean)] / sum_k(x_k - x_mean)^2, b_t = y_mean - a_t × x_mean; then the absolute depth is obtained by D_abs_t = alpha × (a_t × D_rel_t + b_t).

[0045] Step S75: Calculate the depth gradient g(u,v)=sqrt(( D_abs_t(u,v) / u)^2+( D_abs_t(u,v) / v)^2), where sqrt is the square root operation. D_abs_t(u,v) / u represents the rate of change of depth along the horizontal direction of the image. D_abs_t(u,v) / v represents the rate of change of depth along the longitudinal direction of the image. When g(u,v) > threshold τ_depth, the pixel is considered to be in a depth discontinuity region, which together with the segmentation mask boundary forms an edge risk mask.

[0046] Step S8, 2D mask to 3D semantic point cloud mapping: First, perform 2D morphological erosion on the mask, and then back-project the 2D mask pixels to the world coordinate system according to the eroded mask, absolute depth and camera pose, aggregate them into instance point cloud clusters according to global instance number, and generate a global color point cloud map and a point cloud segmentation map. In this embodiment, step S8 specifically includes: Step S81: For the mask Mask_t(i) with obj_id=i in frame t, first perform two-dimensional morphological erosion to obtain Mask_eroded, then take the pixel (u,v) and its absolute depth D_abs_t(u,v) within Mask_eroded, and back-project it to the camera coordinate system in combination with the camera intrinsic parameter K: P_cam=D_abs_t(u,v)×K^(-1)×[u,v,1]^T, where P_cam represents the coordinates of the three-dimensional point corresponding to the pixel in the current camera coordinate system; where u is the horizontal coordinate after mapping to the preprocessed image, v is the vertical coordinate after mapping to the preprocessed image, and the mask and depth map correspond pixel by pixel according to this coordinate.

[0047] Specifically, the two-dimensional morphological erosion is: Mask_eroded = Mask_t(i) E, where E is the structuring element, and Mask_eroded represents the mask obtained by eroding Mask_t(i) through the structuring element E; the structuring element is a 3×3 or 5×5 convolution kernel. This indicates an erosion operation, which removes several pixels from the mask edges, especially the edge areas that overlap with the depth risk mask, thereby reducing the number of depth-skipping pixels entering the 3D point cloud from the source. The size of the erosion kernel is adaptively selected according to the target area. Small targets use smaller kernels to avoid being over-cut, while large targets use larger kernels to improve boundary stability. Step S82: Transform P_cam to a unified world coordinate system using the metric pose C_abs_t=(R_t,t_abs_t): P_world=R_t×P_cam+t_abs_t, where P_world is the world coordinate point corresponding to the pixel; Step S83: Accumulate the P_world values ​​of the same obj_id across all frames to form an instance point cloud cluster PC_i; accumulate all RGB-D points that are not filtered by instance to form a global color point cloud map PC_scene; accumulate all points with obj_id color and semantic labels to form a point cloud segmentation map PC_semantic; where RGB-D indicates that the same pixel has both RGB color value and absolute depth value; wherein, the point cloud segmentation map PC_semantic has semantic labels and obj_id color, used to query the point cloud range of a single object, the global color point cloud map PC_scene is used to generate the scene geometry model, and the instance point cloud cluster PC_i is used for 3D bounding box and semantic map generation.

[0048] Step S9, Dual Denoising and 3D Bounding Box Calculation: Outlier removal is performed on the instance point cloud clusters, and 3D bounding boxes are calculated based on the denoised instance point cloud clusters; In this embodiment, step S9 specifically includes: Step S91: For each point p_k in the instance point cloud cluster PC_i, calculate the average distance d_k between it and its k nearest neighbors. Then calculate the mean μ and standard deviation σ_d of all d_k. If d_k > μ + λ × σ_d, then p_k is determined to be a statistical outlier and removed. Here, λ is the standard deviation multiple of the outlier determination threshold, which is used to control the strictness of denoising: when λ is small, the threshold is low and more points are removed; when λ is large, the threshold is high and more points are retained. It can be taken as 1.0 to 2.0 according to the point cloud density and noise level. This statistical outlier removal method can filter out isolated flying points caused by depth estimation anomalies, mask edge errors and multi-frame registration errors.

[0049] Step S92: Calculate the 3D bounding box for the denoised PC_i. If axisymmetric bounding box (AABB) is used, calculate x_min, x_max, y_min, y_max, z_min, and z_max respectively, with center coordinates of ((x_min+x_max) / 2, (y_min+y_max) / 2, (z_min+z_max) / 2) and dimensions of (x_max-x_min, y_max-y_min, z_max-z_min). Here, x_min is the minimum value of the point cloud along the x-axis, x_max is the maximum value, y_min is the minimum value, y_max is the maximum value, z_min is the minimum value, and z_max is the maximum value. If the target is placed obliquely and AABB generates an empty envelope, then PCA (Principal Component Allocation) is used. Principal Component Analysis (OCB) is performed along the principal axis and the extreme values ​​of each axis are taken in the principal axis coordinate system to form an OBB. For example, for each point p=(x,y,z) in PC_i, calculate x_min=min(x), x_max=max(x), y_min=min(y), y_max=max(y), z_min=min(z), and z_max=max(z) to obtain the six boundary faces, center, and dimensions of the AABB. If the target is roughly parallel to the world coordinate axis, or if path planning requires a fast and conservative occupancy range, use AABB; if targets such as steel bars, long poles, and templates are placed at an angle and AABB will produce a large empty envelope, use PCA to find the principal axis direction and take the extreme values ​​of each axis in the principal axis coordinate system to form an OBB. The number of point clouds is the number of 3D points contained in the denoised instance point cloud cluster, i.e., |PC_i|; the frame range is obtained by the first and last valid mask frame numbers recorded in the global instance table.

[0050] Step S10, Semantic Map, Point Cloud Map and GLB Model Output: Output semantic map, point cloud map and GLB model for use by tower crane path planning, obstacle avoidance and digital twin modules.

[0051] In this embodiment, step S10 specifically includes: Step S101: Output three types of results, namely semantic map, point cloud map, and GLB model, as detailed below: (1) The semantic map is stored in JSON, GeoJSON or database table format. Each object record contains at least obj_id, category name, natural language semantic description, first and last appearance frames, three-dimensional center coordinates, three-dimensional size, bounding box type, number of point clouds, confidence level, color encoding and index relationship with the original image frame. The semantic map can be queried by the tower crane scheduling system, for example, to determine whether there are scaffolding, steel bar piles, personnel or temporary fences near the planned path of the hook.

[0052] (2) The point cloud map includes a global color point cloud map and a point cloud segmentation map with obj_id color and semantic labels; the point cloud map is used to display the geometric structure of the construction site, and the semantic point cloud is used to locate individual targets and calculate obstacle boundaries.

[0053] (3) The GLB model is generated from point cloud through TSDF fusion, Poisson surface reconstruction or system meshing, and records at least scene nodes and their transformations, mesh vertices, triangular face indices, normal vectors, materials or colors, unit scale and world coordinate system. When instance semantics need to be preserved, obj_id, category name and its association with semantic map records are recorded in node name or additional fields. The GLB model preserves the line, surface and obstacle boundaries of the building scene, and can import ROS Navigation, RRT*, A*, MPC or custom tower crane hook path planning modules to generate obstacle avoidance paths, verify the passage space of suspended objects and assist in digital twin visualization.

[0054] Step S102: Provide the semantic map, point cloud map, and GLB model to the tower crane decision module, planning module, and execution module. The decision module reads the semantic map to obtain the 3D bounding boxes of the target object and obstacles. The planning module calculates the travel path of the hook from the starting point to the target point in the GLB or voxel map. The execution module converts the planned path into tower crane slewing, luffing, and hoisting control commands. If the scan finds that an obstacle has been added or its position has changed, the path verification is retried by incrementally updating the semantic map.

[0055] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A general scene semantic mapping method for tower cranes in architectural scenes, characterized in that, Includes the following steps: Step 1: Acquire the original image sequence by using an RGB camera at the end of the tower crane boom or near the tower crane's field of view as the tower crane rotates, and then preprocess it to obtain a preprocessed image sequence; Step 2: Select valid frame preprocessed images and key frame preprocessed images from the preprocessed image sequence; Step 3: Input the preprocessed keyframe image and / or preprocessed effective frame image into the multimodal large model, and execute the processing and inference process of the multimodal large model. If necessary, perform adaptation training in advance, and obtain the output result of the multimodal large model. Step 4: Calculate the comprehensive matching score between the current candidate target and the historical target list based on the output results of the multimodal large model, assign or inherit the global instance number according to the comprehensive matching score, and re-identify and recover targets that have been lost for a short time; Step 5: Use the 2D bounding box, global instance number, and point hints as input to the segmentation model, generate a mask, and bind the global instance number with a unique color to generate a color segmentation map; Step 6: When a new global instance number appears, start a new round of segmentation and tracking with the current frame preprocessed image as the new initial frame preprocessed image, and merge the masks from multiple rounds into a global instance table according to the global instance number; Step 7: Perform relative depth estimation and camera pose estimation on the preprocessed image sequence, and convert the relative depth into absolute depth; Step 8: First, perform two-dimensional morphological erosion on the mask. Then, based on the eroded mask, absolute depth, and camera pose, back-project the two-dimensional mask pixels to the world coordinate system, aggregate them into instance point cloud clusters according to the global instance number, and generate a global color point cloud map and a point cloud segmentation map. Step 9: Remove outliers from the instance point cloud clusters and calculate the 3D bounding box based on the denoised instance point cloud clusters; Step 10: Output semantic map, point cloud map and GLB model for use by tower crane path planning, obstacle avoidance and digital twin modules.

2. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 1, characterized in that, Step 1 specifically includes: Step 11: Use RGB cameras fixed at the front end of the tower crane boom, below the boom, or at the top of the tower crane to collect the original image sequence as the tower crane rotates. If necessary, use drones to collect supplementary images from the blind spots of the tower crane's view to add to the original image sequence. Step 12: Determine the frame extraction interval Δt based on the tower crane's slewing angular velocity ω, the camera's horizontal field of view φ_h, and the target overlap rate ρ_0, satisfying Δ_t=(1-ρ_0)φ_h / ω. If the video frame rate is f, then the frame extraction step size L is taken as an approximate integer of f×Δ_t. Based on the frame extraction step size L, obtain each original image frame from the original image sequence by extracting one frame every L. Step 13: Select a checkerboard, AprilTag, or on-site calibration board as the calibration object. After capturing the calibration object with an RGB camera to obtain a calibration image, detect the calibration feature points in the image. Use the calibration image and feature points to obtain the camera intrinsic parameters K and distortion coefficients. The camera intrinsic parameters K include the focal length fx of the RGB camera along the x-axis, the focal length fy of the RGB camera along the y-axis, the x-coordinate cx of the principal point where the camera optical axis intersects the image plane, and the y-coordinate cy of the principal point in the image pixel coordinate system. The distortion coefficients include radial distortion k1, k2, and k3, and tangential distortion p1 and p2. For a normalized pixel point (x, y) in the camera coordinate system, let r... 2 =x 2 +y 2 The distorted coordinates are: xd=x(1+k1×r 2 +k2×r 4 +k3×r 6 )+2p1xy+p2(r 2 +2x 2 ), yd=y(1+k1×r 2 +k2×r 4 +k3×r 6 )+p1(r 2 +2y2)+2p2xy, Sampling coordinates corresponding to the original image =fx×xd+cx, =fy×yd+cy, and according to =fx×xd+cx and =fy×yd+cy establishes a reverse mapping table to remap the original image to a distortion-free image; where... Represents the sampled x-coordinate in the original image. Represents the sampled ordinate in the original image; Step 14: Project the calibrated feature points back to the original image according to the camera intrinsic parameter K and distortion coefficient, and calculate the average reprojection error; if the average reprojection error exceeds the preset error threshold, re-acquire the calibration image of the calibration object selected in Step 13 and its calibration feature points and recalibrate. After the calibration is qualified, generate the distortion-free image according to Step 13; then proceed to Step 15. Specifically, the calculation of the average reprojection error is as follows: Assume each calibration image has N_cal calibration feature points, and the i-th... l The coordinates of the calibrated feature points are: The predicted projection coordinates are calculated from the known 3D coordinates of the calibrated feature points, the extrinsic parameters of the corresponding calibrated image, the camera intrinsic parameters, and the distortion coefficients. Then the average reprojection error e_rep=(1 / N_cal)×Σ[ l =1 to N_cal]sqrt(( - )^2+( - )^2); where N_cal is the total number of calibrated feature points, l For indexing feature points, The actual number detected in the calibration image l The x-coordinates of each calibrated feature point; The actual number detected in the calibration image l The ordinates of the calibrated feature points; For the first l The projected abscissa of each calibrated feature point; For the first l The projected ordinates of the calibrated feature points; sqrt represents the square root; Step 15: Normalize the brightness of the distorted image using CLAHE adaptive histogram equalization or adaptive gamma correction, and correct it according to I_norm = 255·(I / 255)^gamma, where I_norm is the corrected pixel intensity and I is the input pixel intensity of the distorted image; let the average brightness of the current frame's distorted image be I_mean and the target average brightness be I_target, then gamma = ln(I_target / 255) / ln(I_mean / 255), and limit gamma within the preset upper and lower limits; Step 16: Scale the brightness-normalized image to a uniform resolution, which is selected according to the processing requirements; if the horizontal scaling ratio is s_x and the vertical scaling ratio is s_y, then multiply fx and cx in the camera intrinsic parameters K of the distorted image by s_x, and multiply fy and cy by s_y. Step 17: The original image sequence is processed sequentially through lens distortion correction, brightness normalization, and resolution unification to obtain a preprocessed image sequence, denoted as {I1,I2,...,IN}. The timestamp, crane slewing angle, camera intrinsic parameters, and preprocessing parameters of each preprocessed image frame are recorded. The preprocessing parameters include distortion coefficients, gamma required for brightness normalization, CLAHE window and upper and lower limits, unified resolution, and corresponding horizontal and vertical scaling ratios.

3. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 1, characterized in that, Step 2 specifically includes: Step 21: Perform sharpness check, exposure check and minimum overlap rate check on each frame of the preprocessed image sequence, and retain the images that pass all checks as valid frame preprocessed images to form a set of valid frame preprocessed images V. Step 22: Calculate the number of matching points m_t and overlap rate ρ_t=m_t / min(n_t,n_t-1) between the current frame preprocessed image and the previous keyframe preprocessed image using SuperPoint, SIFT, LoFTR or LightGlue feature matching methods, where n_t and n_t-1 are the number of scene feature points extracted and passed quality screening in the current frame preprocessed image and the previous keyframe preprocessed image, respectively. The matching point determination rules include descriptor matching and geometric verification: First, perform nearest neighbor or bidirectional nearest neighbor search on the two frames of descriptors, retain candidate points whose descriptor distance meets the preset distance threshold or nearest neighbor ratio condition, then use RANSAC to estimate the homography matrix, essential matrix or fundamental matrix, and eliminate candidate points that do not meet the geometric constraints. The finally retained candidate points are the matching points, and m_t is the number of matching points. Step 23: If ρ_t is lower than the overlap threshold, or the rotation angle difference between the current frame preprocessed image and the previous keyframe preprocessed image is greater than the angle threshold, then proceed to step 24. Step 24: Let A be the set of instances or candidate regions already covered by the current keyframe preprocessed image set K, and B_t be the set of candidate regions detected by the current frame preprocessed image. When the proportion of newly added regions Delta_t = Area(B_t-A) / Area(A∪B_t) is greater than the threshold η, the current frame preprocessed image is determined as a keyframe preprocessed image and added to the keyframe preprocessed image set K; where Area represents the number of pixels covered by the current keyframe preprocessed image set, and ∪ represents the union.

4. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 1, characterized in that, Step 3 specifically includes: Step 31: Input the preprocessed keyframe images and / or preprocessed valid frame images into a pre-trained multimodal large model. The multimodal large model is a Qwen-VL, InternVL, GPT-4V type model or a locally deployed visual language model. Perform open vocabulary detection using construction scene cue words. The construction scene cue words require the multimodal large model to identify building materials, construction equipment, temporary facilities, personnel, safety protection devices, floor beams, columns, walls, suspended objects, and obstacles. Output a temporary target number, category name, natural language description, two-dimensional bounding box, and semantic confidence for each target. The natural language description is used to describe appearance features, including color and material features. The specific process is as follows: (1) When processing multimodal large models, the input image is divided and encoded into visual features by the visual encoder, the text encoder converts the prompts of the construction scene into text features, the cross-modal attention module establishes the association between the target region and the category words and descriptive words, the visual localization head outputs a two-dimensional bounding box, the language decoding part outputs the temporary target number, category name and natural language description, and the detection head or similarity result gives the semantic confidence. (2) If on-site proprietary component identification requires adaptation, lightweight adaptation training is performed in advance based on the pre-trained multimodal large model; if no adaptation is required, the pre-trained multimodal large model is directly used to enter the inference stage. The training data consists of tower crane on-site images, manually labeled category names, two-dimensional bounding boxes, natural language descriptions, and cross-frame IDs of the same target. The training method adopts LoRA fine-tuning or instruction fine-tuning to enable the multimodal large model to learn the correspondence between the target description, two-dimensional location, and semantic category of the construction site. (3) In the inference stage, the multimodal big model maps visual features and text features to the same semantic space and generates candidate targets through similarity calculation and visual localization head. For candidate target j, its semantic confidence s_j is: s_j=cos(E_img(region_j),E_text(prompt_j)), where region_j is the j-th candidate image region, E_img(region_j) represents the image region encoding, prompt_j is the j-th text prompt compared with the candidate image region, E_text(prompt_j) represents the text encoding, and cos represents the cosine similarity. When s_j exceeds the preset confidence threshold, the candidate target is output and a detection record set D_t is formed. Each detection record includes temporary target number, category name, natural language description, two-dimensional bounding box, semantic confidence and appearance features.

5. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 1, characterized in that, Step 4 specifically includes: Step 41: For historical target i and current candidate target j, calculate the comprehensive matching score Score(i,j) as follows: Score(i,j)=w1×cos(e_i,e_j)+w2×IoU(B_i^pred,B_j)+w3×cos(a_i,a_j)+w4×exp(-||q_i^pred-q_j|| / σ); where i is the global instance number of the historical target, j is the candidate target number in the preprocessed image of the current frame, e_i is the semantic description embedding of historical target i, e_j is the semantic description embedding of current candidate target j, and IoU represents the intersection area and union area of ​​the two two-dimensional bounding boxes. The ratio of the area of ​​the set, B_i^pred is the 2D bounding box predicted by the camera pose and historical trajectory of the current frame preprocessed image, B_j is the currently detected 2D bounding box, a_i is the appearance feature of the historical target i, a_j is the appearance feature of the current candidate target j, q_i^pred is the position of the historical target i in the image or preliminary 3D space, q_j is the position of the current candidate target j in the image or preliminary 3D space, ||q_i^pred-q_j|| represents the Euclidean distance between q_i^pred and q_j, σ is the spatial distance normalization parameter; w1 to w4 are the corresponding weights and w1+w2+w3+w4=1; Step 42: During matching, construct a score matrix and use the Hungarian algorithm or a greedy strategy to obtain a one-to-one correspondence. When Score(i,j) is greater than the threshold θ_match, the current candidate target j inherits the obj_id of the historical target i. obj_id represents the unique global instance number within the scanning cycle. When the comprehensive matching score of all candidate targets is less than the threshold θ_match, it is determined as a new target and a new obj_id is assigned. Step 43: If a historical target does not appear in the current frame of the preprocessed image, it enters a short-term loss state and records missing_count. missing_count is the number of consecutive frames in which the historical target has not been matched in the current frame of the preprocessed image. When missing_count does not exceed M frames of preprocessed images and subsequent frames of preprocessed images show candidate targets with semantic and spatial matching scores exceeding the re-identification threshold θ_reid, the original obj_id is restored. When missing_count exceeds M or the candidate target leaves the known field of view, the trajectory of the candidate target is terminated. Step 44: Output the detection set G_t with obj_id. Each obj_id is unique throughout the entire scanning period and is bound to the natural language semantic description, historical occurrence frames, and two-dimensional bounding box sequence.

6. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 5, characterized in that, Step 5 specifically includes: Step 51: Take the preprocessed image I_t as the image input, and take the two-dimensional bounding box, obj_id, and point cue generated by the center of the two-dimensional bounding box or the historical mask as the cue input to the segmentation model; Step 52: The image encoder of the segmentation model extracts the image embedding from the preprocessed image I_t. The cue encoder encodes the two-dimensional bounding box or point cue into a cue embedding. The mask decoder fuses the image embedding and the cue embedding and outputs the mask Mask_t(i) and the mask confidence, where i is the global instance number of the historical target. If the mask resolution output by the segmentation model is different from that of the preprocessed image, it is mapped back to the preprocessed image using u=scale_x×u_low, v=scale_y×v_low, where u is the x-coordinate mapped to the preprocessed image, v is the y-coordinate mapped to the preprocessed image; scale_x is the horizontal scaling ratio, scale_y is the vertical scaling ratio; u_low is the x-coordinate in the low-resolution mask, v_low is the y-coordinate in the low-resolution mask; scale_x=W / W_low, scale_y=H / H_low, W is the width of the preprocessed image, H is the height of the preprocessed image, W_low is the width of the low-resolution mask, and H_low is the height of the low-resolution mask. Step 53: Bind a unique color to each obj_id. The color is generated by looking up a table or a hash function. Establish an obj_id-color mapping table and initialize a blank image of the same size as the preprocessed image. Write the pixels in Mask_t(i) with the color corresponding to the obj_id to generate a color segmentation image S_t. Step 54: Establish the correspondence between the mask and obj_id, and assign the pixels whose Euclidean distance between the pixel color in the color segmentation image S_t and the color in the obj_id-color mapping table is less than the threshold τ_color to the corresponding obj_id.

7. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 5, characterized in that, Step 6 specifically includes: Step 61: When it is determined that there is a new obj_id in the current frame preprocessing image, the current frame preprocessing image is used as the new initial frame preprocessing image of the target, and a new round of SAM3 segmentation tracking is started using its two-dimensional bounding box. The mask sequence obtained in this round of tracking is added to the global segmentation result. Step 62: During merging, use obj_id as the primary key to unify the masks, mask confidence and time ranges from different rounds into the global instance table. The time range is determined by the frame number of the first valid mask generated for the same obj_id and the frame number of the last valid mask generated. Step 63: If the same pixel is covered by multiple target masks, retain the target with the highest P among the targets covering the pixel according to priority P = β1×c_mask+β2×c_track+β3×c_sem; if P is the same, retain the target with the smallest obj_id, where c_mask is the mask confidence, c_track is the normalized comprehensive matching score, c_sem is the cosine similarity of the semantic description embedding, and β1, β2 and β3 are the corresponding weights and β1+β2+β3=1; Step 64: If the IoU of two masks exceeds the threshold θ_merge and the semantic description similarity exceeds the threshold θ_sem, then they are considered as the same target and their trajectories are merged; if the IoU of two masks does not exceed the threshold θ_merge or the semantic description similarity does not exceed the threshold θ_sem, then they are kept as different instances, and the color segmentation map S_t of each frame, the mask sequence of each instance, and the global instance table are output.

8. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 5, characterized in that, Step 7 specifically includes: Step 71: Input the preprocessed image I_t into the monocular depth estimation model, and output the relative depth D_rel_t(u,v) after passing through the DINOv2 / ViT encoder and DPT decoder. If necessary, distillation or fine-tuning can be performed in advance using field data with depth or SfM sparse depth. Otherwise, the pre-trained monocular depth estimation model can be used directly for inference. Step 72: For adjacent preprocessed images I_t and I_t+1, extract feature points, estimate the essential matrix E by combining the camera intrinsic parameters K, and obtain the relative rotation and translation directions by E decomposition. Then, optimize the camera pose C_t=(R_t,t_t) for each frame through PnP, triangulation and bundle adjustment, where R_t represents the rotation component from the camera coordinate system to the world coordinate system, and t_t represents the relative translation component from the camera coordinate system to the world coordinate system. The world coordinate system is selected as the origin near the camera pose of the first frame or the center of rotation of the tower crane. Step 73: Extract the camera optical center O_t from the camera pose of each frame, and obtain the estimated rotation center O_c and estimated radius R_est=(1 / N)·Σ_t||O_t-O_c|| through least squares circle fitting, where N is the number of effective camera optical centers that participate in circle fitting and are removed by outlier elimination. The actual horizontal distance R_real from the tower crane camera to the rotation center is measured on site, and then the tower crane scale factor α=R_real / R_est; Step 74: Linearly align the relative depth D_rel_t(u,v) with the relative depth obtained from SfM sparse points or pose triangulation to obtain D_align_t(u,v) = a_t × D_rel_t(u,v) + b_t, where D_align_t(u,v) is the depth aligned to the SfM reconstruction scale, and a_t and b_t are obtained by least-squares fitting of sparse depth points in the same preprocessed image frame. Then, use the crane scale factor to restore the absolute depth D_abs_t(u,v): D_abs_t(u,v) = α × D_align_t(u,v); At the same time, with the origin of the world coordinate system as the scaling center, convert the relative translation component t_t to the absolute translation component t_abs_t: t_abs_t = α × t_t, while keeping the rotation component R_t unchanged, to obtain the metric pose C_abs_t = (R_t, t_abs_t); Step 75: Calculate the depth gradient g(u,v) = sqrt(( / )^2+( / v)^2), where sqrt is the square root operation. / This represents the rate of change of depth along the horizontal direction of the image. / v represents the rate of change of depth along the longitudinal direction of the image. When g(u,v) > threshold τ_depth, the pixel is considered to be in a depth discontinuity region, which together with the segmentation mask boundary forms an edge risk mask.

9. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 8, characterized in that, Step 8 specifically includes: Step 81: For the mask Mask_t(i) of obj_id=i in frame t, first perform two-dimensional morphological erosion to obtain Mask_eroded, then take the pixel (u,v) and its absolute depth D_abs_t(u,v) within Mask_eroded, and back-project it to the camera coordinate system in combination with the camera intrinsic parameter K: P_cam=D_abs_t(u,v)×K^(-1)×[u,v,1]^T, where P_cam represents the coordinates of the three-dimensional point corresponding to the pixel in the current camera coordinate system; where u is the abscissa after mapping to the preprocessed image, and v is the ordinate after mapping to the preprocessed image; Specifically, the two-dimensional morphological erosion is: Mask_eroded = Mask_t(i) E, where E is the structuring element, and Mask_eroded represents the mask obtained by eroding Mask_t(i) through the structuring element E; the structuring element is a 3×3 or 5×5 convolution kernel. This indicates an erosion operation, where the size of the erosion core is adaptively selected based on the target area. Step 82: Transform P_cam to the unified world coordinate system using the metric pose C_abs_t=(R_t,t_abs_t): P_world=R_t×P_cam+t_abs_t, where P_world is the world coordinate point corresponding to the pixel; Step 83: Accumulate the P_world values ​​of the same obj_id across all frames to form an instance point cloud cluster PC_i; accumulate all RGB-D points that are not filtered by instance to form a global color point cloud map PC_scene; accumulate all points with obj_id color and semantic labels to form a point cloud segmentation map PC_semantic; where RGB-D indicates that the same pixel has both RGB color value and absolute depth value.

10. The general scene semantic mapping method for tower cranes in architectural scenes as described in claim 1, characterized in that, Step 9 specifically includes: Step 91: For each point p_k in the instance point cloud cluster PC_i, calculate the average distance d_k between it and its k nearest neighbors. Then calculate the mean μ and standard deviation σ_d of all d_k. If d_k > μ + λ × σ_d, then p_k is determined to be a statistical outlier and removed. Here, λ is the standard deviation multiple of the outlier determination threshold, which is used to control the strictness of noise reduction and is taken as 1.0 to 2.

0. Step 92: Calculate the 3D bounding box for the denoised PC_i. If the axisymmetric bounding box AABB is used, calculate x_min, x_max, y_min, y_max, z_min, and z_max respectively, with the center coordinates as ((x_min+x_max) / 2, (y_min+y_max) / 2, (z_min+z_max) / 2) and the dimensions as (x_max-x_min, y_max-y_min, z_max-z_min). Here, x_min is the minimum value of the point cloud in the x-axis direction, x_max is the maximum value of the point cloud in the x-axis direction, y_min is the minimum value of the point cloud in the y-axis direction, y_max is the maximum value of the point cloud in the y-axis direction, z_min is the minimum value of the point cloud in the z-axis direction, and z_max is the maximum value of the point cloud in the z-axis direction. If the target is placed obliquely and AABB generates an empty envelope, then the PCA principal axis direction is used, and the extreme values ​​of each axis in the principal axis coordinate system are taken to form the OBB. Step 10 specifically includes: Step 101: Output three types of results: semantic map, point cloud map, and GLB model, as detailed below: (1) The semantic map is stored in JSON, GeoJSON or database table format. Each object record contains at least obj_id, category name, natural language semantic description, first and last appearance frames, three-dimensional center coordinates, three-dimensional size, bounding box type, number of point clouds, confidence level, color encoding and index relationship with the original image frame; (2) The point cloud map includes a global color point cloud map and a point cloud segmentation map with obj_id color and semantic labels; (3) The GLB model is generated by point cloud through TSDF fusion, Poisson surface reconstruction or system meshing, and at least the scene nodes and their transformations, mesh vertices, triangular face indices, normal vectors, materials or colors, unit scale and world coordinate system are recorded. When it is necessary to retain instance semantics, the obj_id, category name and its association with the semantic map record are recorded in the node name or additional fields. Step 102: Provide the semantic map, point cloud map, and GLB model to the tower crane decision module, planning module, and execution module. The decision module reads the semantic map to obtain the 3D bounding boxes of the target object and obstacles. The planning module calculates the travel path of the hook from the starting point to the target point in the GLB or voxel map. The execution module converts the planned path into tower crane slewing, luffing, and hoisting control commands. If the scan finds that an obstacle has been added or its position has changed, the path verification is retried by incrementally updating the semantic map.

Citation Information

Patent Citations

  • Three-dimensional model segmentation method and system, computing equipment and electronic equipment

    CN121259020A

  • Large-scene monitoring video abnormal event early warning method based on multi-modal large model

    CN121305463A

  • Water area monitoring method and system based on semantic driving and time sequence behavior cognition

    CN121999352A