Methods, systems, equipment and media for detecting the progress and deviation correction of bridge pier construction

Abstract

The present invention discloses a method, system, equipment and medium for detecting the progress and deviation correction of bridge pier construction, which belongs to the field of image detection, and aims to solve the technical problems of low accuracy and low efficiency of construction deviation correction detection in the existing bridge pier construction process. In the constructed bridge pier detection segmentation model, the YOLO10 target detection network performs target detection on the image, and the CUNet target segmentation network segments the image; the bridge piles under construction are photographed by a drone and the current coordinates and the up, down, left and right deflection angles of the drone are obtained; the image photographed by the drone is input into the bridge pier detection segmentation model and the bridge pier segmentation result is obtained, the segmented image is moved to the target coordinate system, and the center line is projected to obtain the size of each side, and the distance between the end points A and B and the side deviation angle of the center line are compared; the present invention can accurately detect the construction progress of the bridge piles under construction and whether the bridge piles have side deviation.

Description

Technical Field

[0001] The present invention belongs to the technical field of image detection, and relates to bridge pier construction progress detection and deviation correction, and in particular to a method for bridge pier construction progress detection and deviation correction using artificial intelligence technology. Background Art

[0002] Bridge engineering is a crucial control project in line engineering. Construction supervision is an important part of bridge construction technology, the key to ensuring the progress and quality of bridge construction, and the safety guarantee of bridge construction. As a key link in the bridge structure, the construction progress and quality of the bridge will directly affect the construction progress and efficiency of the entire road.

[0003] In recent years, with the rapid development of the low-altitude economy and the acceleration of digitalization and informatization, more and more projects have used drone technology, digitalization and informatization technology to detect, track and manage the progress and quality of bridge construction during bridge construction.

[0004] The invention patent application with application number 202410621811.7 discloses a bridge construction quality supervision method based on a small multi-rotor drone. The MaskR-CNN model is used to detect and identify photos of bridge piers and bridge structures, and the category, bounding box and segmentation mask of each detected object are output. This supervision method can clearly and completely display the three-dimensional structural information of the bridge through the collection of data from a small multi-rotor drone, and perform pixel-level segmentation and bridge sequence structure prediction on the bridge structure through computer vision technology and deep learning technology. It can also timely discover problems and risks in construction through real-time data transmission and analysis.

[0005] The invention patent with application number 202410464498.0 also discloses a knowledge-driven method and device for automatic tracking of bridge structure construction progress. The improved Yolo V5S model is used to identify and extract features from regularly acquired construction monitoring videos, and the automatic tracking of construction progress is achieved based on the comparative analysis of the updated models of each construction stage and the bridge BIM design model.

[0006] The above invention patent can realize feature extraction of bridges through network models such as CNN network and Yolo network. However, in the process of bridge pier construction, in addition to monitoring its construction progress, the construction correction of the bridge pier is also particularly important. The existing construction correction detection is mainly assisted by equipment such as plumb bobs and total stations. First, these detection equipment are inconvenient to use and expensive. Second, they are easily affected by construction scene facilities and subjective interference of personnel. This will seriously affect the detection accuracy and efficiency of construction correction. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problems of low construction correction detection accuracy and low detection efficiency in the existing bridge pier construction process, and to provide a method, system, equipment and medium for detecting the construction progress and correction of bridge piers. By using relevant data obtained during inspection by patrol drones, and with the assistance of lightweight neural network models, spatial geographic information and specific algorithms, efficient and accurate detection of the construction progress and correction of bridge piers can be achieved.

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A method for detecting the progress and deviation correction of bridge pier construction comprises the following steps:

[0010] Step S1, obtaining sample data;

[0011] Obtain sample images of bridge piles under construction, and annotate the bridge piers in the sample images of bridge piles to obtain label data;

[0012] Step S2, constructing a bridge pier detection segmentation model;

[0013] Construct a bridge pier detection and segmentation model, which includes the YOLO10 target detection network and the CUNet target segmentation network. The YOLO10 target detection network is used to detect bridge piers in bridge pile images, and the CUNet target segmentation network is used to segment bridge piers in bridge pile images.

[0014] Step S3, training a bridge pier detection segmentation model;

[0015] The bridge pile sample image and label data obtained in step S1 are used to train the bridge pier detection segmentation model constructed in step S2;

[0016] Step S4, real-time detection and correction;

[0017] Install a camera, gyroscope, and RTK positioning equipment on the drone, and establish a coordinate system from any side point of the bridge pile. The positive direction of the x-axis of the coordinate system is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis of the coordinate system is vertically downward; the drone flies along the positive direction of the x-axis of the coordinate system, and takes images of the bridge piles along the way, and records the current coordinates of the drone when taking each bridge pile image, as well as the up, down, left, and right deflection angles;

[0018] The photographed bridge pile image is input into the bridge pier detection and segmentation model, and the bridge pier detection and segmentation model outputs the bridge pier segmentation result; the segmented bridge pier is rotated according to the deflection angle of the drone when the corresponding bridge pile image is taken, and finally the center line of the segmented bridge pier and its two endpoints A and B are determined; the center line is translated and placed in the xz plane of the coordinate system, with endpoint B located on the x-axis, and endpoint A is vertically projected onto the x-axis to obtain point C; by comparing the pixel-level height between endpoints AB and the lateral deviation angle of the center line before and after, the construction progress and correction results can be obtained.

[0019] Further, in step S2, the CUNet target segmentation network includes a first down-sampling module, a second down-sampling module, a third down-sampling module, a fourth down-sampling module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a fourth up-sampling module and a multi-layer perceptron; the first down-sampling module includes a linear embedding layer and a Swin Transformer sub-module, and each of the remaining down-sampling modules and up-sampling modules includes a Swin Transformer sub-module; the Swin Transformer sub-module of the first down-sampling module and the Swin Transformer sub-module of the fourth up-sampling module, the Swin Transformer sub-module of the second down-sampling module and the Swin Transformer sub-module of the third up-sampling module, and the Swin Transformer sub-module of the third down-sampling module and the Swin Transformer sub-module of the second up-sampling module are all connected through a jump connection module based on compression and excitation, and the Swin Transformer sub-module of the first up-sampling module, the Swin Transformer sub-module of the second up-sampling module, the Swin Transformer sub-module of the third up-sampling module, and the Swin The Transformer submodules are connected in series as the input of the multi-layer perceptron.

[0020] Furthermore, in step S3, when training the CUNet target segmentation network, the loss function is:

[0021]

[0022] in, Represents the true category label of the pixel, Represents the predicted category label of the pixel, Represents any pixel point of the segmented object, W, H, and C represent the width, height, and number of channels of the segmented color image, respectively.

[0023] Furthermore, in step S4, the gyroscope on the drone has an identification deflection range of ±5°, a resolution of 0.0008°, an accuracy of 0.001°, and an output frequency of up to 100 Hz.

[0024] Furthermore, in step S4, when the drone takes the bridge pile image, if the current position is not completely in the positive direction of the x-axis according to the positioning of the RTK positioning device on the drone, the photo closest to the positive direction of the x-axis is selected as the input of the bridge pier detection segmentation model.

[0025] Furthermore, in step S4, when calculating the sideslip angle of the center line, the specific method is:

[0026] Calculate the length of OC based on the coordinates of endpoint A, and the length of OB based on the scale marked on the x-axis for endpoint B; and calculate the length of BC based on the lengths of OC and OB;

[0027] The angle of angle ABC is calculated according to cos(angle ABC) = BC / AB, and then the sideslip angle of the centerline is 90° − angle ABC.

[0028] A bridge pier construction progress and deviation correction detection system, comprising:

[0029] The sample data acquisition module is used to acquire a sample image of a bridge pile and mark the bridge piers in the sample image of the bridge pile to obtain label data;

[0030] A bridge pier detection and segmentation model construction module is used to construct a bridge pier detection and segmentation model. The bridge pier detection and segmentation model includes a YOLO10 target detection network and a CUNet target segmentation network. The YOLO10 target detection network is used to detect bridge piers in bridge pile images, and the CUNet target segmentation network is used to segment bridge piers in bridge pile images.

[0031] A bridge pier detection segmentation model training module is used to train the bridge pier detection segmentation model constructed by the bridge pier detection segmentation model construction module using the bridge pile sample images and label data obtained by the sample data acquisition module;

[0032] The real-time detection and correction module is used to install a camera, a gyroscope, and an RTK positioning device on the drone, and establish a coordinate system from any side point of the bridge pile. The positive direction of the x-axis of the coordinate system is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis of the coordinate system is vertically downward. The drone flies along the positive direction of the x-axis of the coordinate system, and takes images of the bridge piles along the way, and records the current coordinates of the drone when taking each bridge pile image, as well as the up, down, left, and right deflection angles;

[0033] The photographed bridge pile image is input into the bridge pier detection and segmentation model, and the bridge pier detection and segmentation model outputs the bridge pier segmentation result; the segmented bridge pier is rotated according to the deflection angle of the drone when the corresponding bridge pile image is taken, and finally the center line of the segmented bridge pier and its two endpoints A and B are determined; the center line is translated and placed in the xz plane of the coordinate system, with endpoint B located on the x-axis, and endpoint A is vertically projected onto the x-axis to obtain point C; by comparing the pixel-level height between endpoints AB and the lateral deviation angle of the center line before and after, the construction progress and correction results can be obtained.

[0034] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0035] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0036] The beneficial effects of the present invention are as follows:

[0037] In the present invention, firstly, a drone is used to regularly photograph the bridge piles under construction, and the current coordinates of the drone and the up, down, left, and right deflection angles are synchronously obtained during the photographing, so as to provide basic data for the subsequent transformation and calculation of the segmented images; secondly, the constructed bridge pier detection and segmentation model includes a YOLO10 target detection network and a CUNet target segmentation network. The YOLO10 target detection network can perform target detection and identify bridge piers in the images taken by the drone, and the CUNet target segmentation network uses a jump connection module based on compression and excitation for connection, so as to reduce the loss of feature information in the downsampling process, and the importance of each channel is different for different channels. The interrelated pier features are weighted so that the model pays more attention to the piers under construction in the channel and ignores the feature information of the non-pier parts. The multi-stage upsampling information is connected in series through a stage-by-stage manner, which can eliminate the non-feature redundant information generated by the expansion of the image block in the upsampling and achieve more effective image segmentation. The segmented image is then moved to the target coordinate system, and the center line is projected to obtain the size of each side, and the distance between the endpoints A and B is compared and the side deviation angle of the center line is calculated. Finally, the technical problems of low construction correction detection accuracy and low detection efficiency during the construction of the piers are solved, and efficient and accurate detection of the construction progress and correction of the piers is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the process of the present invention;

[0039] Figure 2 It is a schematic diagram of the structure of the CUNet target segmentation network in the present invention;

[0040] Figure 3 It is a structural schematic diagram of a skip connection module based on compression and excitation in the present invention;

[0041] Figure 4 It is a schematic diagram of the deflection of an image taken by a drone due to camera deflection and before and after correction in the present invention;

[0042] Figure 5 It is a schematic diagram of the direction and bridge piers photographed by the drone in the present invention;

[0043] The white line is the center line of the image. The bridge piles on the left are obviously biased to the left, and the bridge piles on the right are obviously biased to the right. The bridge piles near the center line appear to be vertically upward.

[0044] Figure 6 It is a schematic diagram of calculating the deflection angle of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0046] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0047] Example 1

[0048] This embodiment provides a method for detecting the progress and deviation correction of bridge pier construction, which is used to effectively detect and supervise the progress and quality of bridge pier construction (i.e., whether the bridge pier has lateral deviation, etc.). Figure 1 As shown, the specific steps include:

[0049] Step S1, obtaining sample data;

[0050] Obtain sample images of bridge piles under construction, and annotate the bridge piers in the sample images of bridge piles to obtain label data.

[0051] The accurate geographical location of each pier of some bridges under construction and the direction of the bridges are obtained from the bridge designers or bridge builders, and at least 5,000 images of various types of bridge piles under construction are used as sample images of bridge piles. Professional engineers annotate the sample images of bridge piles obtained above, annotate the bridge piers in the images, and obtain label data.

[0052] After obtaining the above-mentioned bridge pile sample images, existing data preprocessing methods may be used for preprocessing, such as cropping, rotation and other preprocessing methods.

[0053] Step S2, constructing a bridge pier detection segmentation model;

[0054] A bridge pier detection and segmentation model is constructed. The bridge pier detection and segmentation model includes the YOLO10 target detection network and the CUNet target segmentation network. The YOLO10 target detection network is used to detect bridge piers in bridge pile images, and the CUNet target segmentation network is used to segment bridge piers in bridge pile images.

[0055] The YOLO10 target detection network can adopt the existing YOLO10 network, which can realize efficient detection of bridge pier targets based on real-time end-to-end target detection technology.

[0056] like Figure 2 As shown, the CUNet target segmentation network includes a first down-sampling module, a second down-sampling module, a third down-sampling module, a fourth down-sampling module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a fourth up-sampling module and a multi-layer perceptron; the first down-sampling module includes a linear embedding layer and a Swin Transformer submodule, and each of the remaining down-sampling modules and up-sampling modules includes a Swin Transformer submodule; the Swin Transformer submodule of the first down-sampling module and the Swin Transformer submodule of the fourth up-sampling module, the Swin Transformer submodule of the second down-sampling module and the Swin Transformer submodule of the third up-sampling module, and the Swin Transformer submodule of the third down-sampling module and the Swin Transformer submodule of the second up-sampling module are all connected through a skip connection module based on compression and excitation, and the Swin Transformer submodule of the first up-sampling module, the Swin Transformer submodule of the second up-sampling module, the Swin Transformer submodule of the third up-sampling module, and the Swin Transformer submodule of the fourth up-sampling module are connected in series as the input of the multi-layer perceptron.

[0057] like Figure 3As shown, the skip connection module based on compression and excitation includes a global pooling layer, a fully connected (FC) layer, a ReLU activation function, and a Sigmoid activation function; taking the skip connection module based on compression and excitation between the Swin Transformer submodule of the first downsampling module and the Swin Transformer submodule of the fourth upsampling module as an example, the output of the Swin Transformer submodule of the first downsampling module passes through the global pooling layer, the fully connected layer, and is activated by the ReLU activation function, and then passes through the fully connected layer in sequence and the Sigmoid function to be output to the Swin Transformer submodule of the fourth upsampling module.

[0058] After the image is divided into blocks, it is input into the first down-sampling module. In the first down-sampling module, the image passes through the linear embedding layer and the Swin Transformer submodule successively, and then the feature map is output; the feature map output by the first down-sampling module is input into the second down-sampling module, and the image blocks are first merged (i.e., feature map merged) in the second down-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third down-sampling module and the fourth down-sampling module are the same as the second down-sampling module, and the description is not repeated here. The output of the Swin Transformer submodule of the fourth down-sampling module is used as the input of the Swin Transformer submodule of the first up-sampling module, and the output of the Swin Transformer submodule of the first up-sampling module is used as the input of the second up-sampling module. The feature map output by the first up-sampling module is input into the second up-sampling module, and the image blocks are first expanded in the second up-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third up-sampling module and the fourth up-sampling module are the same as the second up-sampling module, and the description is not repeated here. The four-level upsampling results (i.e., the outputs of the first upsampling module, the second upsampling module, the third upsampling module, and the fourth upsampling module) are connected in series as the input of a multilayer perceptron, and the multilayer perceptron outputs the segmentation result.

[0059] Step S3, training a bridge pier detection segmentation model;

[0060] The bridge pile sample images and label data obtained in step S1 are used to train the bridge pier detection segmentation model constructed in step S2.

[0061] When training the YOLO10 target detection network, the existing conventional training method can be used, and its loss function can also use the existing loss function.

[0062] When training the CUNet target segmentation network, the loss function is:

[0063]

[0064] in, Represents the true category label of the pixel, Represents the predicted category label of the pixel, Represents any pixel point of the segmented object, W, H, and C represent the width, height, and number of channels of the segmented color image, respectively.

[0065] In the process of target segmentation, any pixel in the image can be divided into two categories. If it is a background point, it is set to black, and if it is a bridge pile pixel, it is set to white. The cross entropy loss function is to sum the loss function CELoss by pixel to get the total loss value of the segmentation. Obviously, this loss function is only used to measure the accuracy of segmentation. When the segmentation result and the annotation result are exactly the same, the loss function should converge to 0 when the segmentation is completed.

[0066] Step S4, real-time detection and correction;

[0067] Install a camera, gyroscope, and RTK positioning equipment on the drone, and establish a coordinate system from any side point of the bridge pile. The positive direction of the x-axis of the coordinate system is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis of the coordinate system is vertically downward; the drone flies along the positive direction of the x-axis of the coordinate system, and takes images of the bridge piles along the way, and records the current coordinates of the drone when taking each bridge pile image, as well as the up, down, left, and right deflection angles;

[0068] The photographed bridge pile image is input into the bridge pier detection segmentation model, and the bridge pier detection segmentation model outputs the bridge pier segmentation result; the segmented bridge pier is rotated according to the deflection angle of the drone when the corresponding bridge pile image is photographed, and finally the center line of the segmented bridge pier and its two endpoints A and B are determined, as shown in Figure 6 As shown in the figure, the center line is translated and placed in the xz plane of the coordinate system, with endpoint B on the x-axis, and endpoint A is vertically projected onto the x-axis to obtain point C; by comparing the pixel-level height between endpoints AB and the side deviation angle of the center line, the construction progress and deviation correction results can be obtained. It should be noted that, of course, the bridge pile image can also be rotated according to the deflection angle of the drone, and then the rotated bridge pile image can be input into the bridge pier detection segmentation model to finally obtain the segmented bridge pile.

[0069] This embodiment uses a drone for intelligent progress detection and correction. The gyroscope on the drone has a recognition deflection range of ±5°, a resolution of 0.0008°, an accuracy of 0.001°, and an output frequency of up to 100Hz. The gyroscope is used to measure the up, down, left, and right deflection angles of the drone at every moment (at least the gyroscope is needed to measure the deflection angle of the drone when shooting bridge piles). The deflection of the camera on the drone causes the image to deflect, so the image needs to be corrected according to the deflection angle of the drone, such as Figure 4As shown, the light blue rectangle shows the deflection and the correction result after rotation. It should be noted that the relationship between the up, down, left, and right deflection angles measured by the drone gyroscope and the corrected rotation of the image can be obtained according to the prior art, that is, according to the up, down, left, and right deflection angles measured by the drone gyroscope, those skilled in the art can know the corrected rotation angle of the image.

[0070] The RTK positioning device on the drone (based on differential GPS high-precision positioning equipment) can display the exact distance (positive and negative error is 1.5cm) that the drone deviates from the specified route up, down, left, and right, thereby determining the dynamic coordinate system of the drone at a certain moment (that is, at least the dynamic coordinate system of the drone when taking the photo needs to be obtained). Since the streaming results of the RTK positioning device can more accurately reflect the current position coordinates of the drone, the current coordinate system should be determined according to the RTK positioning results when each photo was taken during calculation to reduce calculation errors.

[0071] The drone establishes a coordinate system from any side point of the bridge pile (taking this point as the coordinate origin (0,0,0)), such as Figure 6 As shown, the positive direction of the x-axis is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis is vertically downward. The drone flies along the positive direction of the x-axis to the position point (x1,0,0), and takes photos along the way. The x-axis unit is converted from GPS length to pixel unit. The starting point position of the drone is obtained by the RTK positioning device it carries, and the position during flight is recorded in each corresponding image taken.

[0072] Since the location of the center point of each bridge pile is known (obtained from the designer or builder), the photo of the center point of the bridge pile taken by the aircraft is taken out and tested (where, Figure 5 The deviation of the bridge piles that are not on the center line of the photo is displayed). Due to the jitter of the drone during flight, the current position measured by the RTK positioning device may not be on the preset straight line, so the photo closest to the center position can be selected; that is, when the drone takes the image of the bridge pile, if the current position measured by the RTK positioning device on the drone is not completely in the positive direction of the x-axis, the photo closest to the positive direction of the x-axis is selected as the input of the bridge pier detection segmentation model.

[0073] When calculating the sideslip angle of the center line, the specific method is:

[0074] Calculate the center line of the split bridge pier and translate the center line to the xz plane, with the origin at O. The upper end point is compared with the x-axis at point B. Draw a straight line from the lower breakpoint A of the center line to the x-axis perpendicularly intersecting the x-axis at point C, connecting OA. Calculate the length of OC based on the coordinates of endpoint A, and the length of OB based on the scale marked on the x-axis of endpoint B (the length of OB can be read from the scale marked on the X-axis in pixels); the length of BC is obtained based on the length of OC and the length of OB; the angle of angle ABC is calculated based on cos(angle ABC)=BC / AB, and then the side deviation angle of the center line is 90°-angle ABC.

[0075] The pixel-level height of the bridge piers is calculated for the segmented line AB during daily inspections. By comparing before and after, the progress of the bridge pile construction and whether there is any lateral deviation of the construction bridge piles can be accurately detected, so as to reasonably arrange the construction and correct the deviation in time.

[0076] Example 2

[0077] This embodiment provides a pier construction progress and deviation correction detection system, including:

[0078] The sample data acquisition module is used to acquire sample images of bridge piles and mark the bridge piers in the sample images of bridge piles to obtain label data.

[0079] The accurate geographical location of each pier of some bridges under construction and the direction of the bridges are obtained from the bridge designers or bridge builders, and at least 5,000 images of various types of bridge piles under construction are used as sample images of bridge piles. Professional engineers annotate the sample images of bridge piles obtained above, annotate the bridge piers in the images, and obtain label data.

[0080] After obtaining the above-mentioned bridge pile sample images, existing data preprocessing methods may be used for preprocessing, such as cropping, rotation and other preprocessing methods.

[0081] The pier detection and segmentation model building module is used to build a pier detection and segmentation model. The pier detection and segmentation model includes a YOLO10 target detection network and a CUNet target segmentation network. The YOLO10 target detection network is used to detect piers in bridge pile images, and the CUNet target segmentation network is used to segment piers in bridge pile images.

[0082] The YOLO10 target detection network can adopt the existing YOLO10 network, which can achieve efficient detection of bridge pier targets based on real-time end-to-end target detection technology without creative labor.

[0083] like Figure 2As shown, the CUNet target segmentation network includes a first down-sampling module, a second down-sampling module, a third down-sampling module, a fourth down-sampling module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a fourth up-sampling module and a multi-layer perceptron; the first down-sampling module includes a linear embedding layer and a Swin Transformer submodule, and each of the remaining down-sampling modules and up-sampling modules includes a Swin Transformer submodule; the Swin Transformer submodule of the first down-sampling module and the Swin Transformer submodule of the fourth up-sampling module, the Swin Transformer submodule of the second down-sampling module and the Swin Transformer submodule of the third up-sampling module, and the Swin Transformer submodule of the third down-sampling module and the Swin Transformer submodule of the second up-sampling module are all connected through a skip connection module based on compression and excitation, and the Swin Transformer submodule of the first up-sampling module, the Swin Transformer submodule of the second up-sampling module, the Swin Transformer submodule of the third up-sampling module, and the Swin Transformer submodule of the fourth up-sampling module are connected in series as the input of the multi-layer perceptron.

[0084] like Figure 3 As shown, the skip connection module based on compression and excitation includes a global pooling layer, a fully connected (FC) layer, a ReLU activation function, and a Sigmoid activation function; taking the skip connection module based on compression and excitation between the Swin Transformer submodule of the first downsampling module and the Swin Transformer submodule of the fourth upsampling module as an example, the output of the Swin Transformer submodule of the first downsampling module passes through the global pooling layer, the fully connected layer, and is activated by the ReLU activation function, and then passes through the fully connected layer in sequence and the Sigmoid function to be output to the Swin Transformer submodule of the fourth upsampling module.

[0085] After the image is divided into blocks, it is input into the first down-sampling module. In the first down-sampling module, the image passes through the linear embedding layer and the Swin Transformer submodule successively, and then the feature map is output; the feature map output by the first down-sampling module is input into the second down-sampling module, and the image blocks are first merged (i.e., feature map merged) in the second down-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third down-sampling module and the fourth down-sampling module are the same as the second down-sampling module, and the description is not repeated here. The output of the Swin Transformer submodule of the fourth down-sampling module is used as the input of the Swin Transformer submodule of the first up-sampling module, and the output of the Swin Transformer submodule of the first up-sampling module is used as the input of the second up-sampling module. The feature map output by the first up-sampling module is input into the second up-sampling module, and the image blocks are first expanded in the second up-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third up-sampling module and the fourth up-sampling module are the same as the second up-sampling module, and the description is not repeated here. The four-level upsampling results (i.e., the outputs of the first upsampling module, the second upsampling module, the third upsampling module, and the fourth upsampling module) are connected in series as the input of a multilayer perceptron, and the multilayer perceptron outputs the segmentation result.

[0086] The bridge pier detection segmentation model training module is used to train the bridge pier detection segmentation model constructed by the bridge pier detection segmentation model construction module using the bridge pile sample images and label data obtained by the sample data acquisition module.

[0087] When training the YOLO10 target detection network, the existing conventional training method can be used, and its loss function can also use the existing loss function without creative work.

[0088] When training the CUNet target segmentation network, the loss function is:

[0089] in, Represents the true category label of the pixel, Represents the predicted category label of the pixel, Represents any pixel point of the segmented object, W, H, and C represent the width, height, and number of channels of the segmented color image, respectively.

[0090] In the process of target segmentation, any pixel in the image can be divided into two categories. If it is a background point, it is set to black, and if it is a bridge pile pixel, it is set to white. The cross entropy loss function is to sum the loss function CELoss by pixel to get the total loss value of the segmentation. Obviously, this loss function is only used to measure the accuracy of segmentation. When the segmentation result and the annotation result are exactly the same, the loss function should converge to 0 when the segmentation is completed.

[0091] The real-time detection and correction module is used to install a camera, a gyroscope, and an RTK positioning device on the drone, and establish a coordinate system from any side point of the bridge pile. The positive direction of the x-axis of the coordinate system is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis of the coordinate system is vertically downward. The drone flies along the positive direction of the x-axis of the coordinate system, and takes images of the bridge piles along the way, and records the current coordinates of the drone when taking each bridge pile image, as well as the up, down, left, and right deflection angles;

[0092] The photographed bridge pile image is input into the bridge pier detection segmentation model, and the bridge pier detection segmentation model outputs the bridge pier segmentation result; the segmented bridge pier is rotated according to the deflection angle of the drone when the corresponding bridge pile image is photographed, and finally the center line and two endpoints A and B of the segmented bridge pier are determined, such as Figure 6 As shown in the figure, the center line is translated and placed in the xz plane of the coordinate system, with endpoint B on the x-axis, and endpoint A is vertically projected onto the x-axis to obtain point C; by comparing the pixel-level height between endpoints AB and the side deviation angle of the center line, the construction progress and deviation correction results can be obtained. It should be noted that, of course, the bridge pile image can also be rotated according to the deflection angle of the drone, and then the rotated bridge pile image can be input into the bridge pier detection segmentation model to finally obtain the segmented bridge pile.

[0093] This embodiment uses a drone for intelligent progress detection and correction. The gyroscope on the drone has a recognition deflection range of ±5°, a resolution of 0.0008°, an accuracy of 0.001°, and an output frequency of up to 100Hz. The gyroscope is used to measure the up, down, left, and right deflection angles of the drone at every moment (at least the gyroscope is needed to measure the deflection angle of the drone when shooting bridge piles). The deflection of the camera on the drone causes the image to deflect, so the image needs to be corrected according to the deflection angle of the drone, such as Figure 4 As shown, the light blue rectangle shows the deflection and the correction result after rotation. It should be noted that the relationship between the up, down, left, and right deflection angles measured by the drone gyroscope and the corrected rotation of the image can be obtained according to the prior art, that is, according to the up, down, left, and right deflection angles measured by the drone gyroscope, those skilled in the art can know the corrected rotation angle of the image.

[0094] The RTK positioning device on the drone (based on differential GPS high-precision positioning equipment) can display the exact distance (positive and negative error is 1.5cm) that the drone deviates from the specified route up, down, left, and right, thereby determining the dynamic coordinate system of the drone at a certain moment (that is, at least the dynamic coordinate system of the drone when taking the photo needs to be obtained). Since the streaming results of the RTK positioning device can more accurately reflect the current position coordinates of the drone, the current coordinate system should be determined according to the RTK positioning results when each photo was taken during calculation to reduce calculation errors.

[0095] The drone establishes a coordinate system from any side point of the bridge pile (taking this point as the coordinate origin (0,0,0)), such as Figure 6 As shown, the positive direction of the x-axis is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis is vertically downward. The drone flies along the positive direction of the x-axis to the position point (x1,0,0), and takes photos along the way. The x-axis unit is converted from GPS length to pixel unit. The starting point position of the drone is obtained by the RTK positioning device it carries, and the position during flight is recorded in each corresponding image taken.

[0096] Since the location of the center point of each bridge pile is known (obtained from the designer or builder), the photo of the center point of the bridge pile taken by the aircraft is taken out and tested (where, Figure 5 The deviation of the bridge piles that are not on the center line of the photo is displayed). Due to the jitter of the drone during flight, the current position measured by the RTK positioning device may not be on the preset straight line, so the photo closest to the center position can be selected; that is, when the drone takes the image of the bridge pile, if the current position measured by the RTK positioning device on the drone is not completely in the positive direction of the x-axis, the photo closest to the positive direction of the x-axis is selected as the input of the bridge pier detection segmentation model.

[0097] When calculating the sideslip angle of the center line, the specific method is:

[0098] Calculate the center line of the split bridge pier and translate the center line to the xz plane, with the origin at O. The upper end point is compared with the x-axis at point B. Draw a straight line from the lower breakpoint A of the center line to the x-axis perpendicularly intersecting the x-axis at point C, connecting OA. Calculate the length of OC based on the coordinates of endpoint A, and the length of OB based on the scale marked on the x-axis of endpoint B (the length of OB can be read from the scale marked on the X-axis in pixels); the length of BC is obtained based on the length of OC and the length of OB; the angle of angle ABC is calculated based on cos(angle ABC)=BC / AB, and then the side deviation angle of the center line is 90°-angle ABC.

[0099] The pixel-level height of the bridge piers is calculated for the segmented line AB during daily inspections. By comparing before and after, the progress of the bridge pile construction and whether the bridge piles are lateral deviation can be accurately detected, so as to reasonably arrange the construction and correct the deviation in time.

[0100] Example 3

[0101] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned method for detecting the progress of bridge pier construction and deviation correction.

[0102] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote control, a touch pad, or a voice control device.

[0103] The memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the detection method of the bridge pier construction progress and correction, etc. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0104] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of the computer device. In this embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the detection method for the pier construction progress and deviation correction.

[0105] Example 4

[0106] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned method for detecting the construction progress and deviation correction of piers.

[0107] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor executes the steps of the above-mentioned method for detecting the progress of bridge pier construction and deviation correction.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the method for detecting the progress and deviation correction of the pier construction described in the embodiment of the present application.

[0109] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A method for detecting the progress and deviation correction of bridge pier construction, characterized in that: The following steps are involved: Step S1, obtaining sample data; Obtain a sample image of a bridge pile, and annotate the bridge piers in the sample image of the bridge pile to obtain label data; Step S2, constructing a bridge pier detection segmentation model; Construct a bridge pier detection and segmentation model, which includes the YOLO10 target detection network and the CUNet target segmentation network. The YOLO10 target detection network is used to detect bridge piers in bridge pile images, and the CUNet target segmentation network is used to segment bridge piers in bridge pile images. Step S3, training a bridge pier detection segmentation model; The bridge pile sample image and label data obtained in step S1 are used to train the bridge pier detection segmentation model constructed in step S2; Step S4, real-time detection and correction; Install a camera, gyroscope, and RTK positioning equipment on the drone, and establish a coordinate system from any side point of the bridge pile. The positive direction of the x-axis of the coordinate system is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis of the coordinate system is vertically downward; the drone flies along the positive direction of the x-axis of the coordinate system, and takes images of the bridge piles along the way, and records the current coordinates of the drone when taking each bridge pile image, as well as the up, down, left, and right deflection angles; The photographed bridge pile image is input into the bridge pier detection and segmentation model, and the bridge pier detection and segmentation model outputs the bridge pier segmentation result; the segmented bridge pier is rotated according to the deflection angle of the drone when the corresponding bridge pile image is photographed, and finally the center line of the segmented bridge pier and its two endpoints A and B are determined; the center line is translated and placed in the xz plane of the coordinate system, with endpoint B located on the x-axis, and endpoint A is vertically projected onto the x-axis to obtain point C; the construction progress and correction results are obtained by comparing the pixel-level height between endpoints AB and the side deviation angle of the center line before and after; Specifically, the CUNet target segmentation network includes a first down-sampling module, a second down-sampling module, a third down-sampling module, a fourth down-sampling module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a fourth up-sampling module, and a multi-layer perceptron; the first down-sampling module includes a linear embedding layer and a Swin Transformer submodule, and the second down-sampling module, the third down-sampling module, the fourth down-sampling module, the first up-sampling module, the second up-sampling module, the third up-sampling module, and the fourth up-sampling module all include a Swin Transformer submodule; the Swin Transformer submodule of the first down-sampling module and the Swin Transformer submodule of the fourth up-sampling module, the Swin Transformer submodule of the second down-sampling module and the Swin Transformer submodule of the third up-sampling module, and the Swin Transformer submodule of the third down-sampling module and the Swin Transformer submodule of the second up-sampling module are all connected through a jump connection module based on compression and excitation; the Swin Transformer submodule of the first up-sampling module, the Swin Transformer submodule of the second up-sampling module, and the Swin The outputs of the Transformer submodule and the Swin Transformer submodule of the fourth upsampling module are connected in series as the input of the multilayer perceptron; The image is input into the first down-sampling module after being divided into blocks. In the first down-sampling module, the image is successively passed through the linear embedding layer and the Swin Transformer submodule before outputting the feature map; the feature map output by the first down-sampling module is input into the second down-sampling module, and the image blocks are first merged in the second down-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third down-sampling module and the fourth down-sampling module are the same as the second down-sampling module; the output of the Swin Transformer submodule of the fourth down-sampling module is used as the input of the Swin Transformer submodule of the first up-sampling module, and the output of the Swin Transformer submodule of the first up-sampling module is used as the input of the second up-sampling module; the feature map output by the first up-sampling module is input into the second up-sampling module, and the image blocks are first expanded in the second up-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third up-sampling module and the fourth up-sampling module are the same as the second up-sampling module; the outputs of the first up-sampling module, the second up-sampling module, the third up-sampling module and the fourth up-sampling module are connected in series as the input of the multi-layer perceptron, and the multi-layer perceptron outputs the segmentation result; The compression and excitation based jump connection module includes a global pooling layer, a fully connected layer, a ReLU activation function, and a Sigmoid activation function; taking the compression and excitation based jump connection module between the Swin Transformer submodule of the first down-sampling module and the Swin Transformer submodule of the fourth up-sampling module as an example, the output of the Swin Transformer submodule of the first down-sampling module passes through the global pooling layer, the fully connected layer, and is activated by the ReLU activation function, and then passes through the fully connected layer in sequence and the Sigmoid function to be output to the Swin Transformer submodule of the fourth up-sampling module.

2. A method for detecting the progress and deviation correction of bridge pier construction according to claim 1, characterized in that: When training the bridge pier detection segmentation model, the loss function of training the CUNet target segmentation network is: Among them, y t Represents the true category label of the pixel, y p Represents the predicted category label of the pixel, y represents any pixel of the segmented object, W, H, and C represent the width, height, and number of channels of the segmented color image, respectively.

3. A method for detecting the progress and deviation correction of bridge pier construction according to claim 2, characterized in that: The gyroscope on the UAV has an identification deflection range of ±5°, a resolution of 0.0008°, an accuracy of 0.001°, and an output frequency of ≤100 Hz.

4. A method for detecting the progress and deviation correction of bridge pier construction according to claim 3, characterized in that: When the UAV takes a picture of a bridge pile, if the current position is not completely in the positive direction of the x-axis according to the positioning of the RTK positioning device on the UAV, the photo closest to the positive direction of the x-axis is selected as the input of the bridge pier detection segmentation model.

5. A method for detecting the progress and deviation correction of bridge pier construction according to claim 3, characterized in that: The specific calculation method of the sideslip angle of the center line is as follows: The length of OC is calculated according to the coordinates of endpoint A, and the length of OB is obtained according to the marked scale of endpoint B on the x-axis. The length of OB is read from the marked scale on the x-axis in pixels, and O is the origin of the coordinate axis; the length of BC is obtained according to the length of OC and the length of OB; the angle of angle ABC is calculated according to cos(angle ABC)=BC / AB, and then the lateral deviation angle of the centerline is 90°-angle ABC.

6. A bridge pier construction progress and deviation correction detection system, characterized in that: include: The sample data acquisition module is used to acquire a sample image of a bridge pile and mark the bridge piers in the sample image of the bridge pile to obtain label data; A bridge pier detection and segmentation model construction module is used to construct a bridge pier detection and segmentation model. The bridge pier detection and segmentation model includes a YOLO10 target detection network and a CUNet target segmentation network. The YOLO10 target detection network is used to detect bridge piers in bridge pile images, and the CUNet target segmentation network is used to segment bridge piers in bridge pile images. The CUNet target segmentation network includes a first down-sampling module, a second down-sampling module, a third down-sampling module, a fourth down-sampling module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a fourth up-sampling module, and a multi-layer perceptron; the first down-sampling module includes a linear embedding layer and a Swin Transformer submodule, and the second down-sampling module, the third down-sampling module, the fourth down-sampling module, the first up-sampling module, the second up-sampling module, the third up-sampling module, and the fourth up-sampling module all include a Swin Transformer submodule; the Swin Transformer submodule of the first down-sampling module and the Swin Transformer submodule of the fourth up-sampling module, the Swin Transformer submodule of the second down-sampling module and the Swin Transformer submodule of the third up-sampling module, and the Swin Transformer submodule of the third down-sampling module and the Swin Transformer submodule of the second up-sampling module are all connected through a jump connection module based on compression and excitation; the Swin Transformer submodule of the first up-sampling module, the Swin Transformer submodule of the second up-sampling module, and the Swin Transformer submodule of the third up-sampling module The outputs of the Transformer submodule and the Swin Transformer submodule of the fourth upsampling module are connected in series as the input of the multilayer perceptron; The image is input into the first down-sampling module after being divided into blocks. In the first down-sampling module, the image is successively passed through the linear embedding layer and the Swin Transformer submodule before outputting the feature map; the feature map output by the first down-sampling module is input into the second down-sampling module, and the image blocks are first merged in the second down-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third down-sampling module and the fourth down-sampling module are the same as the second down-sampling module; the output of the Swin Transformer submodule of the fourth down-sampling module is used as the input of the Swin Transformer submodule of the first up-sampling module, and the output of the Swin Transformer submodule of the first up-sampling module is used as the input of the second up-sampling module; the feature map output by the first up-sampling module is input into the second up-sampling module, and the image blocks are first expanded in the second up-sampling module, and then the feature map is output after passing through the Swin Transformer submodule; the third up-sampling module and the fourth up-sampling module are the same as the second up-sampling module; the outputs of the first up-sampling module, the second up-sampling module, the third up-sampling module and the fourth up-sampling module are connected in series as the input of the multi-layer perceptron, and the multi-layer perceptron outputs the segmentation result; The skip connection module based on compression and excitation includes a global pooling layer, a fully connected layer, a ReLU activation function, and a Sigmoid activation function; taking the skip connection module based on compression and excitation between the Swin Transformer submodule of the first downsampling module and the Swin Transformer submodule of the fourth upsampling module as an example, the output of the Swin Transformer submodule of the first downsampling module passes through the global pooling layer, the fully connected layer, and is activated by the ReLU activation function, and then passes through the fully connected layer in sequence and is output to the Swin Transformer submodule of the fourth upsampling module through the Sigmoid function; A bridge pier detection segmentation model training module is used to train the bridge pier detection segmentation model constructed by the bridge pier detection segmentation model construction module using the bridge pile sample images and label data obtained by the sample data acquisition module; The real-time detection and correction module is used to install a camera, a gyroscope, and an RTK positioning device on the drone, and establish a coordinate system from any side point of the bridge pile. The positive direction of the x-axis of the coordinate system is parallel to the horizontal plane of the bridge pile, and the positive direction of the y-axis of the coordinate system is vertically downward. The drone flies along the positive direction of the x-axis of the coordinate system, and takes images of the bridge piles along the way, and records the current coordinates of the drone when taking each bridge pile image, as well as the up, down, left, and right deflection angles; The photographed bridge pile image is input into the bridge pier detection and segmentation model, and the bridge pier detection and segmentation model outputs the bridge pier segmentation result; the segmented bridge pier is rotated according to the deflection angle of the drone when the corresponding bridge pile image is taken, and finally the center line of the segmented bridge pier and its two endpoints A and B are determined; the center line is translated and placed in the xz plane of the coordinate system, with endpoint B located on the x-axis, and endpoint A is vertically projected onto the x-axis to obtain point C; the construction progress and correction results are obtained by comparing the pixel-level height between endpoints AB and the lateral deviation angle of the center line.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 5.

Images