Unmanned aerial vehicle image bridge construction progress recognition method based on multi-modal fusion image recognition

Abstract

The present application belongs to the technical field of unmanned aerial vehicle remote sensing monitoring and intelligent image recognition, and aims to solve the technical problems of single feature extraction, insufficient semantic modeling, lack of multi-source data fusion and inaccurate progress quantification in the existing unmanned aerial vehicle image bridge construction progress identification method. Provide a kind of unmanned aerial vehicle image bridge construction progress identification method based on multi-modal fusion image recognition, which integrates deep learning feature extraction, cross-modal data fusion and construction stage semantic modeling technology, collects multi-source image data of the whole process of bridge construction by unmanned aerial vehicle, realizes intelligent identification of the construction state of key components such as bridge foundation, pier column and beam body, evaluates the bridge construction progress based on the three-dimensional layered progress evaluation index system, generates and outputs the report containing the quantitative progress evaluation result and visual annotation, and provides technical support for bridge engineering construction process control, progress early warning and resource optimization configuration.

Description

Technical Field

[0001] This invention belongs to the field of UAV remote sensing monitoring and intelligent image recognition technology, specifically relating to a method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition. Background Technology

[0002] Bridge construction projects have long construction cycles, complex working environments, and diverse component types. Traditional construction progress monitoring relies on manual inspections, total station measurements, and report statistics, which suffer from problems such as low efficiency, data lag, strong subjectivity, and difficulty in reaching high-risk areas.

[0003] With the development of UAV technology and computer vision, construction progress monitoring based on UAV imagery has become a research hotspot. However, existing technologies still have significant limitations: feature extraction is too simplistic, with existing methods relying heavily on texture and contour features of single-modal images, making it difficult to handle complex scenarios such as changes in lighting, occlusion interference, and dynamic changes in component shapes during bridge construction, resulting in insufficient accuracy in identifying key construction components; semantic modeling for construction stages is lacking, as no dedicated semantic model has been built for the temporal characteristics of bridge construction, making it impossible to effectively distinguish different stages of component construction, such as pier reinforcement binding, formwork installation, and concrete pouring, leading to significant errors in progress quantification; multi-source data fusion is insufficient, failing to fully utilize the complementary information from UAV visible light, infrared, and LiDAR multi-modal images, resulting in poor adaptability to complex construction environments; and progress evaluation indicators are limited, often based on the completion rate of individual components, lacking a comprehensive evaluation capability for the connection of construction procedures and the overall progress matching degree.

[0004] These problems make it difficult for existing technologies to meet the needs of refined construction management in bridge engineering, and there is an urgent need to build an intelligent monitoring method that takes into account recognition accuracy, environmental adaptability and scientific progress quantification. Summary of the Invention

[0005] In order to solve at least one of the above-mentioned technical problems in the prior art, the present invention provides a method for identifying bridge construction progress based on UAV images using multimodal fusion image recognition.

[0006] This invention employs the following technical solution: a method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition, comprising the following steps: Image data and 3D point cloud data of the bridge construction area were acquired, and a multimodal dataset was obtained after spatiotemporal registration. A dual-branch feature extraction network is constructed to extract texture, temperature, and spatial geometric features from a multimodal dataset. Based on a cross-modal weighted attention fusion module and an improved feature pyramid network BiFPN, the texture, temperature, and spatial geometric features from the multimodal dataset are deeply fused to generate a fused feature map. By combining a pre-built semantic knowledge base of bridge construction stages, a dynamic detection head is used to identify key components in the fused feature map and determine their construction stage. Based on the recognition results, a three-dimensional hierarchical progress evaluation index system for bridge construction scenarios is constructed. The progress evaluation index includes, but is not limited to, component completion rate, process connection efficiency, and critical path matching degree. Bridge construction progress is assessed based on a three-dimensional hierarchical progress assessment index system, generating and outputting a report that includes quantitative progress assessment results and visual annotations.

[0007] Preferably, the image data of the bridge construction area includes RGB images and infrared images; the process of obtaining the multimodal dataset includes spatial registration using a rigid registration algorithm based on mutual information, eliminating illumination differences through adaptive radiometric correction, filtering out noise using an improved RANSAC algorithm, and thus obtaining a standardized multimodal dataset with spatiotemporal alignment.

[0008] Preferably, in the dual-branch feature extraction network, the first branch processes RGB images and infrared images based on the improved Transformer architecture. The Transformer architecture introduces a fusion mechanism of local window attention and global sparse attention to extract texture and temperature features. The second branch processes 3D point cloud data based on PointPillars and PointNet++ to extract spatial geometric features.

[0009] Preferably, the construction stages include, but are not limited to, foundation construction, reinforcement binding in pier construction, formwork installation, concrete pouring, beam construction, and bridge deck construction; the identification stage of key components and their respective construction stages also includes the introduction of a temporal attention mechanism to construct a temporal evolution model of construction status based on historical multimodal data.

[0010] Preferably, in the progress evaluation index of the three-dimensional hierarchical progress evaluation index system, the component completion rate is included. The formula is used to calculate the ratio of the number of completed construction components to the total number of planned components within the current image. In the formula, To identify the number of completed components, The total number of planned construction components corresponding to this process; process connection efficiency. This formula is used to quantify the degree of connection and completion between adjacent construction processes based on the construction sequence logic of components. In the formula, m This represents the number of currently associated processes. This is the weighting coefficient for process connection. The matching degree of the i-th process connection; the critical path matching degree. The formula used to compare the construction plan network diagram is: Where CP represents the set of all processes along the critical path of the construction plan. To identify the set of completed processes that belong to the critical path, Given the planned duration for process j, calculate the overlap ratio between the current actual construction process and the project's critical path to reflect the overall rationality of the project's progress.

[0011] Preferably, the steps for evaluating bridge construction progress based on a three-dimensional hierarchical progress evaluation index system include: The analytic hierarchy process (AHP) is used to construct a judgment matrix, and the weights corresponding to component completion rate, process connection efficiency, and critical path matching degree are determined through consistency checks. , , ; Calculate the overall actual progress based on weighting ratio. ; Combined with the pre-set schedule The schedule deviation rate is obtained as follows: This allows for the assessment of bridge construction progress.

[0012] Preferably, it also includes the introduction of a time-series smoothing factor. The overall actual progress is optimized; the optimized formula for calculating the overall actual progress is as follows: The optimized overall actual progress is then used in the calculation formula to replace the original overall actual progress.

[0013] Preferably, it also includes periodically or based on newly added multimodal data, using an incremental learning mechanism to fine-tune the feature extraction network and recognition model, and using a category-aware incremental sample screening strategy based on the distribution characteristics of bridge construction component categories to update easily confused component samples, including but not limited to steel bars, formwork, and beams. The model training is optimized using a weighted mixture loss function, the expression of which is: In the formula This is a focal loss, used to reduce the impact of component category imbalance; To improve the weighted intersection and union ratio loss, it is used to optimize the bounding box regression accuracy for the problems of blurred and occluded boundaries of construction components; Redundant convolutional channels are removed by pruning the structured model, and model parameters are compressed using 8-bit integer quantization.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention effectively overcomes interference from complex environments such as lighting and occlusion by using multimodal data fusion and an improved feature extraction network, which can effectively improve the accuracy of key component identification and the classification accuracy during the construction phase. The constructed three-dimensional progress evaluation system takes into account the completion status of components, process connection and overall path matching degree, which is more in line with the actual engineering situation than the traditional single index evaluation method. This invention automates the entire process from image data acquisition, feature extraction, status recognition to progress assessment, which can significantly reduce manual intervention, lower monitoring costs, and improve construction management efficiency. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is an overall flowchart of the UAV image bridge construction progress recognition method based on multimodal fusion image recognition, as per the present invention. Figure 2 This is a structural diagram of the multimodal feature fusion extraction model of the present invention; Figure 3 This is the progress quantitative evaluation index system and calculation logic diagram of the present invention; Figure 4 This is a comparison of bridge construction progress identification results in an embodiment of the present invention (left: original image, middle: feature fusion result, right: progress quantification labeling result). Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that the structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportional relationships, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should fall within the scope of the technical content disclosed in the present invention. It should be noted that in this specification, relational terms such as "first" and "second" are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0019] This invention provides an embodiment: like Figure 1 As shown, a method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition includes the following steps: S1: Acquire image data and 3D point cloud data of the bridge construction area, and obtain a multimodal dataset after spatiotemporal registration.

[0020] In this embodiment, visible light cameras, infrared thermal imagers, and LiDAR equipment mounted on a UAV are used to collect RGB images, infrared images, and 3D point cloud data of the bridge construction area along a preset flight path. Spatial registration is performed on the above multimodal data using a rigid registration algorithm based on mutual information, with the registration error controlled within 1 pixel. Illumination differences are eliminated through adaptive radiometric correction, and noise is filtered out using an improved RANSAC algorithm, thereby obtaining a standardized multimodal dataset with spatiotemporal alignment.

[0021] The RANSAC algorithm is an iterative method used to estimate the parameters of a mathematical model from observation data containing outliers. LiDAR technology is a detection system that integrates laser, GPS, and inertial navigation systems. It determines the target distance by measuring the time difference between laser pulse emission and reception, and fuses three-dimensional spatial positioning data to generate an accurate digital 3D model.

[0022] S2: Construct a dual-branch feature extraction network to extract texture and temperature features and spatial geometric features from the multimodal dataset; based on the cross-modal weighted attention fusion module and the improved feature pyramid network BiFPN, perform deep fusion of texture and temperature features and spatial geometric features from the multimodal dataset to generate a fused feature map.

[0023] In this embodiment, as Figure 2As shown, the first branch processes RGB and infrared images based on an improved Transformer architecture, which introduces a fusion mechanism of local window attention and global sparse attention to extract texture and temperature features; the second branch processes 3D point clouds based on PointPillars and PointNet++ to extract spatial geometric features; the two types of features are deeply fused through a cross-modal weighted attention fusion module and an improved feature pyramid network BiFPN to generate a fused feature map.

[0024] The weight calculation formula for the cross-modal weighted attention fusion module is as follows: ,in The modal feature significance score; This is the attention weight matrix; Represents the Sigmoid activation function; Learnable bias parameters are used to correct feature baseline shifts and improve the model's fitting freedom; the improved BiFPN adds cross-scale attention connections. The improved GSConv module introduces a channel attention mechanism on top of the standard GSConv to enhance the extraction capability of weak texture features.

[0025] Transformer is a deep learning model architecture based on self-attention, consisting of an encoder and a decoder, widely used in natural language processing and sequence-to-sequence tasks. PointPillars is a 3D object detection network whose core idea is to convert 3D point cloud data into 2D pseudo-images, thereby leveraging mature 2D convolutional neural networks for fast object detection. PointNet++ is a deep learning model for processing 3D point cloud data, aiming to address the shortcomings of the original PointNet model in local feature extraction and sparse point cloud processing; by introducing a hierarchical structure and multi-scale feature extraction, it significantly improves the classification, segmentation, and semantic understanding capabilities of point cloud data. BiFPN, Bidirectional Feature Pyramid Network, is a network structure used for multi-scale feature fusion in object detection.

[0026] S3: Combining a pre-built semantic knowledge base of bridge construction stages, a dynamic detection head that integrates scale perception, spatial perception, and task perception attention mechanisms identifies key components in the fused feature map and determines their construction stage.

[0027] In this embodiment, the construction stages include, but are not limited to, foundation construction, rebar tying in pier construction, formwork installation, concrete pouring, beam construction, and bridge deck construction; the identification stage of key components and their respective construction stages also includes the introduction of a temporal attention mechanism to construct a temporal evolution model of construction status based on historical multimodal data.

[0028] S4: Based on the recognition results, construct a three-dimensional hierarchical progress evaluation index system for bridge construction scenarios. The progress evaluation index includes, but is not limited to, component completion rate, process connection efficiency, and critical path matching degree. In this embodiment, the progress evaluation indicators of the three-dimensional hierarchical progress evaluation indicator system include: 1. Component completion rate The formula is used to calculate the ratio of the number of completed construction components to the total number of planned components within the current image. In the formula To identify the number of completed components, This represents the total number of planned construction components corresponding to this process. 2. Process connection efficiency This formula is used to quantify the degree of connection and completion between adjacent construction processes based on the construction sequence logic of components. In the formula, m is the number of currently associated processes. This is the weighting coefficient for process connection. The degree of connection and matching of the i-th process; 3. Critical path matching degree Based on the construction plan network diagram, the formula is: Where CP represents the set of all processes along the critical path of the construction plan. This refers to the set of completed processes that belong to the critical path, identified by S3. Given the planned duration for process j, calculate the overlap ratio between the current actual construction process and the project's critical path to reflect the overall rationality of the project's progress.

[0029] S5: Based on a three-dimensional hierarchical progress assessment index system, bridge construction progress is assessed, and a report containing quantitative progress assessment results and visual annotations is generated and output.

[0030] like Figure 3 As shown, the steps for evaluating bridge construction progress based on a three-dimensional hierarchical progress evaluation index system include: S51: Construct a decision matrix using the Analytic Hierarchy Process (AHP), and determine the weights corresponding to component completion rate, process connection efficiency, and critical path matching degree through a consistency test (CR < 0.1). , , ,satisfy ; S52: Calculate the overall actual progress based on weighting ratio ; S53: Combine with the preset schedule The schedule deviation rate is obtained as follows: This allows for the assessment of bridge construction progress.

[0031] In this embodiment, step S52 further includes introducing a time-series smoothing factor. The value is set to 0.7~0.9 to adapt to the temporal continuity of construction images, optimize the overall actual progress, and suppress progress jumps caused by single-frame recognition errors. The optimized formula for calculating the overall actual progress is as follows: The optimized overall actual schedule is then substituted into the optimized overall actual schedule calculation formula to replace the original overall actual schedule (e.g., Figure 4 As shown, left: original image, middle: feature fusion result, right: progress quantization annotation result.

[0032] S6: This application also includes incremental model learning and optimization. The feature extraction network and recognition model are fine-tuned periodically or according to newly added multimodal data using an incremental learning mechanism. Based on the distribution characteristics of bridge construction component categories, a category-aware incremental sample screening strategy is adopted to update easily confused component samples, including but not limited to steel bars, formwork, and beams. The model training is optimized using a weighted mixture loss function, the expression of which is: In the formula This is a focal loss used to reduce the impact of component category imbalance, where... This represents the confidence probability that the model predicts the current construction component as its true category. ∈(0,1), representing the predicted probability that components such as steel bars, formwork, and box girders are correctly classified; The weighting coefficients for category balance are assigned based on the sample proportions of various bridge components, with larger values ​​allocated to components that are difficult to collect samples, such as embedded parts and small connectors. To address the imbalance in the types of construction components; To focus on modulation factors for difficult examples, γ=2 is optimally selected. The loss contribution weight of easily identifiable components with large areas is reduced, allowing the model training to focus on occluded and incomplete difficult-to-identify components. To improve the weighted intersection and union ratio loss, it is used to optimize the bounding box regression accuracy for the problems of blurred and occluded boundaries of construction components; By pruning redundant convolutional channels in the structured model and using 8-bit integer quantization to compress model parameters, the model size is compressed while maintaining processing efficiency and reducing processing latency, ensuring a low rate of decrease in component recognition accuracy, thus meeting the real-time processing needs of drones or edge devices.

[0033] This invention proposes a multimodal fusion feature extraction architecture. It extracts cross-modal features through an improved Transformer and PointPillars dual-branch network, and combines a weighted attention fusion module with an improved BiFPN to achieve deep fusion of multi-scale features, improving component recognition accuracy in complex environments. It constructs a semantic knowledge base and temporal evolution model for bridge construction stages, introducing a dynamic detection head with a multi-dimensional attention mechanism to achieve fine-grained recognition of construction stages, solving the technical challenge of traditional methods in distinguishing similar construction states. It designs a three-dimensional progress quantification evaluation system including component completion rate, process connection efficiency, and critical path matching degree, combining analytic hierarchy process (AHP) and incremental learning mechanism to achieve accurate quantification and dynamic updating of construction progress. Finally, it proposes a lightweight inference optimization scheme, using hybrid loss function training, model pruning, and quantization techniques to improve processing speed while ensuring recognition accuracy, meeting real-time monitoring requirements.

[0034] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition, characterized in that, Includes the following steps: Image data and 3D point cloud data of the bridge construction area were acquired, and a multimodal dataset was obtained after spatiotemporal registration. A dual-branch feature extraction network is constructed to extract texture, temperature, and spatial geometric features from a multimodal dataset; Based on the cross-modal weighted attention fusion module and the improved feature pyramid network BiFPN, texture and temperature features and spatial geometric features in multimodal datasets are deeply fused to generate fused feature maps. By combining a pre-built semantic knowledge base of bridge construction stages, a dynamic detection head is used to identify key components in the fused feature map and determine their construction stage. Based on the recognition results, a three-dimensional hierarchical progress evaluation index system for bridge construction scenarios is constructed. The progress evaluation index includes, but is not limited to, component completion rate, process connection efficiency, and critical path matching degree. Bridge construction progress is assessed based on a three-dimensional hierarchical progress assessment index system, generating and outputting a report that includes quantitative progress assessment results and visual annotations.

2. The method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition according to claim 1, characterized in that: The image data of the bridge construction area includes RGB images and infrared images. The process of obtaining the multimodal dataset includes spatial registration using a rigid registration algorithm based on mutual information, eliminating illumination differences through adaptive radiometric correction, filtering out noise using an improved RANSAC algorithm, and finally obtaining a standardized multimodal dataset with spatiotemporal alignment.

3. The method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition according to claim 2, characterized in that: In the dual-branch feature extraction network, the first branch processes RGB and infrared images based on the improved Transformer architecture. The Transformer architecture introduces a fusion mechanism of local window attention and global sparse attention to extract texture and temperature features. The second branch uses PointPillars and PointNet++ to process 3D point cloud data and extract spatial geometric features.

4. The method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition according to claim 2, characterized in that: The construction phases include, but are not limited to, foundation construction, reinforcement binding in pier construction, formwork installation, concrete pouring, beam construction, and bridge deck construction; the identification phase of key components and their respective construction phases also includes the introduction of a temporal attention mechanism to construct a temporal evolution model of construction status based on historical multimodal data.

5. The method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition according to claim 2, characterized in that: In the three-dimensional hierarchical progress evaluation index system, the component completion rate is one of the progress evaluation indicators. The formula is used to calculate the ratio of the number of completed construction components to the total number of planned components within the current image. In the formula, in the formula To identify the number of completed components, The total number of planned construction components corresponding to this process; process connection efficiency. Used to quantify the degree of completion of adjacent construction processes based on component-based construction sequence logic. In the formula, m is the number of currently associated processes. This is the weighting coefficient for process connection. The matching degree of the i-th process connection; the critical path matching degree. The formula used to compare the construction plan network diagram is: Where CP represents the set of all processes along the critical path of the construction plan. To identify the set of completed processes that belong to the critical path, Given the planned duration for process j, calculate the overlap ratio between the current actual construction process and the project's critical path to reflect the overall rationality of the project's progress.

6. The method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition according to claim 5, characterized in that: The steps for evaluating bridge construction progress based on a three-dimensional hierarchical progress evaluation index system include: The analytic hierarchy process (AHP) is used to construct a judgment matrix, and the weights corresponding to component completion rate, process connection efficiency, and critical path matching degree are determined through consistency checks. , , ; Calculate the overall actual progress based on weighting ratio. ; Combined with the pre-set schedule The schedule deviation rate is obtained as follows: This allows for the assessment of bridge construction progress.

7. The method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition according to claim 6, characterized in that: It also includes the introduction of a time-series smoothing factor. The overall actual progress is optimized; the optimized formula for calculating the overall actual progress is as follows: The optimized overall actual progress is then used in the calculation formula to replace the original overall actual progress.

8. The method for identifying bridge construction progress from UAV images based on multimodal fusion image recognition according to claim 6, characterized in that: It also includes periodically or based on newly added multimodal data, using an incremental learning mechanism to fine-tune the feature extraction network and recognition model, and using a category-aware incremental sample screening strategy based on the distribution characteristics of bridge construction component categories to update easily confused component samples, including but not limited to steel bars, formwork, and beams. The model training is optimized using a weighted mixture loss function, the expression of which is: In the formula This is a focal loss, used to reduce the impact of component category imbalance; To improve the weighted intersection and union ratio loss, it is used to optimize the bounding box regression accuracy for the problems of blurred and occluded boundaries of construction components; Redundant convolutional channels are removed by pruning the structured model, and model parameters are compressed using 8-bit integer quantization.

Images