Infrastructure construction site management and control method and device based on multi-source data fusion

Abstract

The invention belongs to the technical field of infrastructure site management and control and safety management, and provides an infrastructure site management and control method and device based on multi-source data fusion, and the technical scheme is that operation data of large-scale buildings such as an operation site, a transmission tower and the like are transmitted back, the site construction quality, site equipment and machinery monitoring is realized, and meanwhile, the safety of the large-scale buildings is guaranteed. Data information is analyzed when the two ends of the pole tower are subjected to unbalanced tension under the side sinking condition of the pole tower, the non-uniform sinking condition and the actual condition around the power transmission line are combined, and the inclination state and the local stress state of the pole tower are analyzed according to the inclination data monitored by the power transmission line pole tower inclination monitoring system on site. The method is advantaged in that an actual power system is guided to carry out timely and effective prediction and early warning on pole tower inclination, data support is provided for prevention of power transmission line pole tower collapse, an algorithm combined with deep learning inclination distance measurement is provided in combination with on-site monitoring images, pole tower inclination monitoring is further realized, and result reliability is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrastructure site control and safety management, and in particular relates to an infrastructure site control method and device based on multi-source data fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As an important part of infrastructure site management and control, the safety and stability of power poles are the prerequisites for reliable power transmission. Due to natural disasters, human factors, and the quality of power poles, it is inevitable that they will tilt, deform, or even collapse, causing power outages and affecting life and production. Manual patrol inspections cannot timely detect and eliminate safety hazards. At the same time, most areas where power poles are installed have complex terrain and harsh environments, which makes it more difficult to timely detect and eliminate hidden dangers.

[0004] The existing infrastructure site control method uses the collected coordinate data of the construction points and the tower height data of the construction points to compare with the pre-planned data to control the infrastructure site. It is impossible to predict the relevant status of the equipment at the infrastructure site in advance. At the same time, the prediction result is not reliable based on a simple comparison with the planned data. Summary of the invention

[0005] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides an infrastructure site control method based on multi-source data fusion, which transmits back the operation data of large-scale buildings such as work sites and transmission towers to realize on-site construction quality, on-site equipment and machinery monitoring; secondly, in combination with the actual situation around the transmission line, the online prediction method of the tilt angle based on the dynamic sliding window is analyzed; finally, in combination with the on-site monitoring image, an algorithm combined with deep learning tilt ranging is proposed to further achieve the purpose of monitoring the tilt of the tower and increase the reliability of the results.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a construction site management and control method based on multi-source data fusion, comprising the following steps:

[0008] Obtain the posture data of transmission towers and image data of infrastructure sites;

[0009] The predicted value of the inclination angle of the transmission tower is obtained according to the posture data of the transmission tower and the trained tower inclination angle prediction model; wherein the construction process of the tower inclination angle prediction model is as follows:

[0010] Extracting a transmission tower posture data sequence having correlation from the transmission tower posture data;

[0011] The regression algorithm in machine learning is used to take the posture data sequence of the transmission tower with correlation as the input of the tower tilt angle prediction model, and the predicted value of the transmission tower tilt angle is obtained by fitting.

[0012] Combined with the image data from the infrastructure site, the disparity information is calculated and refined layer by layer using image layers of different resolutions to obtain a fused disparity map. The distance tilt depth corresponding to each pixel is calculated based on the fused disparity map to obtain the actual tilt depth of the transmission tower in three-dimensional space. When the actual tilt depth is greater than the set depth threshold, a tower tilt warning is issued.

[0013] Furthermore, the posture data of the transmission tower includes the forced vibration status of the transmission tower, the inclination angle of the transmission tower, the meteorological data of the transmission tower operation site, the temperature of the transmission wires and cable joints, and the strain status data of the cement tower foundation.

[0014] Furthermore, a dynamic sliding window is used to extract a sequence of posture data of transmission poles with correlation in the posture data of transmission poles, and the posture data of transmission poles with correlation is used as a training sample. The training sample includes two parts: features and labels. The training sample is the posture data of continuous historical transmission poles, and the label is the posture data of the transmission poles that follows the training features.

[0015] Furthermore, the length of the dynamic sliding window is:

[0016]

[0017]

[0018] Among them, p i is the ratio of the i-th data to the sum of the data sequence, is the average value of the data sequence, n is the number of data sequences, {x1,x2…,x n} is the data sequence.

[0019] Furthermore, the tower inclination angle prediction model training specifically includes:

[0020] Pre-training the tower inclination angle prediction model, during which a neural network including multiple restricted Boltzmann machines and a BP neural network is used; each restricted Boltzmann machine includes an explicit layer and a hidden layer, the bottom explicit layer is used as an input layer, the input layer and multiple hidden layers form a deep belief network for extracting features of sample data, and the last hidden layer and the output layer form a BP neural network;

[0021] The overall parameters of the pre-trained tower inclination angle prediction model are reversely fine-tuned, wherein the reverse fine-tuning includes: obtaining the corresponding output according to the input of the pre-trained neural network; recalculating the weights and biases of the deep belief network based on the back-propagation algorithm; and modifying the weights and biases of the deep belief network based on the momentum gradient descent method.

[0022] Furthermore, the predicted value of the tilt angle of the transmission tower is combined with the image data of the infrastructure site, and the disparity information is calculated and refined layer by layer using image layers of different resolutions to obtain a fused disparity map, including:

[0023] A three-layer image pyramid is constructed. The first layer retains the highest resolution image. The second layer performs Gaussian blur processing on the image of the first layer, and then downsamples it to reduce the resolution to 50% of the original image to obtain the second resolution image. The third layer performs Gaussian blur processing on the second layer again to further reduce the image resolution to 50% of the second layer to obtain the third resolution image.

[0024] The disparity calculation is performed independently at each layer of the image pyramid. The calculation process includes:

[0025] The matching cost is calculated based on the similarity between each pixel in the left image and the corresponding disparity position pixel in the right image;

[0026] Aggregate costs along multiple paths through dynamic programming to obtain aggregated costs;

[0027] Select the disparity value with the minimum cost for each pixel from the aggregated cost, traverse all possible disparity values ​​of each pixel, and select the disparity value that minimizes the aggregated cost as the final disparity of the pixel;

[0028] After calculating the disparity of all layers, the results of different layers are fused to obtain a fused disparity map.

[0029] Furthermore, after calculating the disparity of all layers, the results of different layers are fused to obtain a fused disparity map, including:

[0030] Determine the weight basis and resolution of each image layer, where the weight of each image layer is proportional to its resolution;

[0031] Adjust the initial weights according to the confidence of the disparity calculation at each image layer;

[0032] For each pixel p, the weighted average disparity value is calculated using the disparity data of all layers and the corresponding weights.

[0033] A second aspect of the present invention provides an infrastructure site management and control system based on multi-source data fusion, comprising:

[0034] A multi-source data acquisition module, which is used to acquire the posture data of the transmission tower and the image data of the infrastructure site;

[0035] The tilt angle prediction module is used to obtain the predicted value of the tilt angle of the transmission tower according to the posture data of the transmission tower and the trained tower tilt angle prediction model; wherein the construction process of the tower tilt angle prediction model is: extracting the posture data sequence of the transmission tower with correlation from the posture data of the transmission tower; using the regression algorithm in machine learning, taking the posture data sequence of the transmission tower with correlation as the input of the tower tilt angle prediction model, and fitting to obtain the predicted value of the tilt angle of the transmission tower;

[0036] The early warning judgment module is used to combine the image data of the infrastructure site, use image layers of different resolutions to calculate and refine the disparity information layer by layer, and obtain a fused disparity map; based on the fused disparity map, the distance tilt depth corresponding to each pixel is calculated to obtain the actual tilt depth of the transmission tower in three-dimensional space; when the actual tilt depth is greater than the set depth threshold, a tower tilt warning is issued.

[0037] A third aspect of the present invention provides a computer-readable storage medium.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the infrastructure site management and control method based on multi-source data fusion as described above.

[0039] A fourth aspect of the present invention provides a computer device.

[0040] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the infrastructure site control method based on multi-source data fusion as described above are implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention extracts the posture data sequence of the transmission tower with correlation from the posture data of the transmission tower, predicts the inclination angle prediction value of the transmission tower, and combines the image data of the infrastructure site, uses image layers of different resolutions to calculate and refine the disparity information layer by layer, obtains the fused disparity map, and obtains the actual inclination depth of the transmission tower in three-dimensional space; when the actual inclination depth is greater than the set depth threshold, a tower inclination warning is issued. The reliability of the prediction result is increased, which is conducive to the timely control of the infrastructure site.

[0043] 2. Pre-train the prediction model based on the deep belief network, and reversely fine-tune the pre-trained light tower inclination angle prediction model to obtain the final tower inclination angle prediction model for prediction. It can be adjusted in the convergence direction and the adjustment amount in this direction can be accelerated. Therefore, this method has a faster running speed while avoiding network divergence. At the same time, compared with the method in which the momentum factor is a fixed constant, the momentum factor of this method can be continuously adjusted during the training process to speed up the convergence speed.

[0044] 3. Use image layers of different resolutions to calculate and refine the disparity information layer by layer to obtain a fused disparity map. The advantage of this method is that it can quickly capture a wide range of disparity information at lower resolution layers, while focusing on improving the details and accuracy of the disparity at high resolution layers, ensuring the accuracy of subsequent disparity calculations and improving the reliability of tower tilt warnings.

[0045] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 It is a flow chart of a construction site management and control method based on multi-source data fusion provided by an embodiment of the present invention;

[0048] Figure 2 It is the system framework of the transmission tower monitoring device provided by the embodiment of the present invention;

[0049] Figure 3 It is an algorithm flow chart of deep learning tilt ranging provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] Embodiment 1

[0054] like Figure 1 As shown, this embodiment provides a construction site management and control method based on multi-source data fusion, including the following steps:

[0055] Step 1: Collect relevant data on the infrastructure site based on the transmission tower monitoring system built;

[0056] Obtain the forced vibration status of the transmission tower, the tilt status of the transmission tower, the meteorological data (temperature, humidity) at the transmission tower operation site, the temperature of the transmission wire and cable joints, and the strain status of the cement tower foundation; at the same time, cooperate with the video anti-theft monitoring system to make a comprehensive evaluation of the operation status of the transmission tower equipment. By installing acquisition boards with sensors on the transmission tower and other ancillary equipment (transmission wires, cement tower foundation), the operation status of the transmission tower is fully sensed. An on-site regional central base station is installed at the center of the transmission tower monitoring area. Various sensor data are transmitted over short distances. The wireless communication module (Zigbee) uses the wireless 5G network to transmit various data files and analysis results to the remote power grid monitoring center after summarizing and simply analyzing and processing them. Subsequently, the expert software analysis system of the relevant power system conducts in-depth analysis and integration of various data and combines the historical operation data of the transmission tower to conduct a comprehensive assessment of the safe operation level of the transmission tower. When a transmission tower fails, it can be discovered in time and effective warnings can be issued, thereby avoiding serious power production safety accidents caused by transmission tower operation accidents and ensuring the safe and stable operation of the power system.

[0057] The vibration of transmission towers refers to the tendency of the transmission towers to produce certain damping oscillations under the dynamic loads of various excitation sources, thereby causing the internal stress (tension) of the transmission towers to change and move. Due to their outdoor environment and towering large steel frame structure, transmission towers are easily damaged by external objects, theft, and other damages, all of which will cause the towers to vibrate.

[0058] The vibration sensor is installed in the middle of the transmission tower to test the vibration caused by various impacts that the transmission tower may be subjected to. The system needs to judge and analyze the hardness of the damaging object that causes the tower vibration and the degree of contact with the tower, and then conduct a comprehensive assessment of the transmission tower vibration condition based on the monitoring results. The specific measurement parameters are as follows:

[0059] Table 1 Vibration measurement parameters of transmission towers

[0060] Range Frequency measurement range Resolution Temperature range Applications 0～50g 0.2Hz~1000Hz 0.00008g -40℃～+150℃ Large vibration shock measurement

[0061] Analyze the reasons for the tilt of transmission towers and set monitoring parameters

[0062] The tilt sensor is installed on the top of the transmission tower at an angle perpendicular to the transmission tower. When the tower body tilts, the output signal of the tilt sensor will change. The tilt angle of the transmission tower can be calculated based on the output voltage signal value. The measurement parameters are set as follows:

[0063] Table 2 Transmission tower tilt monitoring parameters

[0064] Range Applications Resolution Temperature range -90°～+90° Tower tilt 1° -40℃～+85℃

[0065] like Figure 2 As shown in the figure, the whole system consists of three parts: the underlying acquisition board, the on-site regional central base station and the remote power grid monitoring center. The sensor data information obtained by the acquisition board is sent to the on-site regional central base station in the area through the ZigBee module. Then the on-site regional central base station performs simple analysis and processing on the data on site, and then transmits the relevant data and analysis results together with the on-site video and image files through the 5G wireless network to the remote power grid monitoring center. The system can be used in urban transmission corridors with frequent vehicle traffic and dense population, as well as 35kV to 750kV high-voltage transmission towers with complex natural and geological environments.

[0066] Step 2: Based on the actual situation around the transmission line and the data collected in step 1, the predicted value of the transmission tower inclination angle is obtained based on the trained tower inclination angle prediction model;

[0067] Step 201: extracting a transmission tower posture data sequence having correlation from the posture data of the transmission tower;

[0068] The transmission tower data is collected through a dynamic sliding window. The collected posture data is used to build a prediction model through the regression algorithm in machine learning. The prediction model is used to perform online prediction of the transmission tower posture data. The collected training data set is divided into multiple training samples. The samples are divided into two parts: features and labels. The features of the training samples are a continuous historical data, and the labels of the training samples are the historical data that follows the training features. Assuming that the length of the sliding window is n, the transmission tower uniaxial tilt angle sequence collected using the sliding window is {x1, x2…, x n The sample feature is a continuous tilt angle sequence with a length of k, and the label is the tilt angle data immediately following the sample feature, so the length of a training sample is k+1. When the transmission tower tilt angle sequence collected by the sliding window is correlated, the prediction model can be better fitted.

[0069] Local trend of transmission tower inclination sequence. In order to ensure the relevance of the prediction model input data, a dynamic sliding window data training method is used. The length of the dynamic sliding window is:

[0070]

[0071]

[0072] Among them, p i is the ratio of the i-th data to the sum of the data sequence, is the mean value of the data series,

[0073] Step 202: using a regression algorithm in machine learning, taking a posture data sequence of a transmission tower with correlation as an input to a tower tilt angle prediction model, and fitting to obtain a predicted value of the transmission tower tilt angle; comprising the following steps:

[0074] Step 221: The input data of the model are the data measured by the posture sensor, represented by x1, x2, and x3 respectively. The output of the model is the state prediction represented by y p Indicates that the training set samples {x1,x2,x3} are established.

[0075] Step 222: Pre-training the prediction model, using a neural network including multiple restricted Boltzmann machines and a BP neural network; each restricted Boltzmann machine includes an explicit layer and a hidden layer, the bottom explicit layer is used as an input layer, the input layer and the multiple hidden layers form a deep belief network for extracting features of sample data, and the last hidden layer and the output layer form a BP neural network;

[0076] Assume that the number of neurons in the visible layer and hidden layer are n, m, and v respectively. i ,h jis the state of the i-th neuron in the visible layer and the j-th neuron in the hidden layer, and their corresponding offset is a i ; b j ;w ij is the connection weight between the nth neuron in the explicit layer and the first neuron in the hidden layer; β = (w ij ,a i ,b j ) are the parameters of RBM. Then the energy function of RBM in state (v,h) can be expressed as:

[0077]

[0078] Randomly give the visible layer input v = (v1,…,v i , / v n ), the jth hidden layer activation function;

[0079]

[0080] Given a hidden layer h=(h1…,h i ,…h n ), the first unit of the explicit layer is activated (assigned a value of 1):

[0081]

[0082] Used to solve the RBM parameter β=(w ij ,a i ,b j ), first select a sample as the initial state of the list layer v; then update the state v' = (v') according to h. i ); then calculate the update of the hidden layer neurons h'=(h′ j ), the update formula is:

[0083] Δw ij =σ( <v i h j >- <v i 'h' j >)

[0084] Δa i =σ( <v i >- <v′ i >),

[0085] Δb j =σ( <h j >- <h' j >)

[0086] Among them, σ is the learning rate algorithm, and <·> is the mathematical expectation of the variable.

[0087] Then another sample is selected as the initial state of the display layer, and the above steps are repeated until all samples in the training sample are selected once.

[0088] Step 2023: After the pre-training is completed, it is necessary to reversely fine-tune the overall parameters of the tower inclination angle prediction model so that the model converges to the global optimum. The present invention adds a momentum factor (equivalent to adding a damping term) to reduce the oscillation of the training process. At the same time, the increased momentum factor can make the network ignore smaller surface features, thereby avoiding falling into a local minimum. In addition, the method can accelerate the adjustment of the convergence direction and has a faster running speed. Therefore, the present invention introduces a momentum factor for optimization when updating the network weights, and the fine-tuning process of the model is as follows:

[0089] Step 20231: Obtain the corresponding output according to the input of the model. According to the initial value of the parameter obtained by pre-training, the state of the hidden layer neural unit is determined and its activation value is calculated. Then, it propagates backward layer by layer and calculates the activation value of the neural unit in the hidden layer of each layer. Finally, the output of the output layer (top layer) is calculated.

[0090] Step 20232: The error back propagation algorithm is used to recalculate the weights and biases of the deep belief network. The cost function of the network is as follows:

[0091]

[0092] In the equation, is the number of hidden layer neurons in the qth RBM; is the squared error vector of the network, Y j ', Y j are the actual and ideal outputs (w (q) ,b (q) ) are the weights and bias vectors to be trained in the qth RBM, PV power prediction based on deep belief network with momentum factor.

[0093] Step 20233: The momentum gradient descent method is used to modify the weights and biases of the deep belief network. The correction formula is:

[0094]

[0095] (w ij ,b j )(t+1)=(w j ,b j )+Δ(w ij ,b j )(t+1),

[0096] In the formula, α is the momentum factor learning rate. This method adds αΔ(w ij,b j )(t). Each adjustment to the network weights and offsets will take into account the current network weights and bias adjustments as well as the results of the last network weight adjustment. This method can always make adjustments in the convergence direction and accelerate the amount of adjustments in that direction. Therefore, this method has a faster running speed while avoiding network divergence. At the same time, compared with the method where the momentum factor is a fixed constant, the momentum factor of this method can be continuously adjusted during the training process to speed up the convergence speed.

[0097] Step 3: Calculate the inclination angle of the tower through deep learning algorithms, filter the analysis results in step 2, and issue early warnings to achieve on-site control.

[0098] In step 3, the prediction results in step 2 are used to calculate the inclination angle of the tower through a deep learning algorithm, and the analysis results in step 2 are screened to provide early warning and achieve on-site control. The specific steps include:

[0099] For the image video collected in step 1, the binocular ranging method based on multi-scale disparity calculation uses image layers of different resolutions to calculate and refine disparity information layer by layer, and then merges this information into a high-precision disparity map through an intelligent fusion algorithm, and finally converts it into a depth map. The advantage of this method is that it can quickly capture a wide range of disparity information at lower resolution layers, while focusing on improving the details and accuracy of the disparity at high resolution layers. Figure 3 As shown, the following steps are included:

[0100] Step 301: Camera Calibration

[0101] Use a chessboard-based stereo calibration method. Shoot the chessboard calibration plate from multiple angles, use the feature point detection algorithm to identify the intersections of the chessboard, and use these intersections as control points to calculate the internal and relative external parameters of the camera. The internal parameters include focal length and optical center, and the external parameters cover the relative position and rotation angle between cameras. Set different viewing angles and distances, shoot multiple sets of chessboard pictures, use processing software to estimate parameters, and output accurate calibration data.

[0102] Step 302: Image Capture and Correction

[0103] The main purpose of the capture and correction step is to ensure that high-quality images are acquired synchronously from the binocular camera system and to perform necessary corrections on these images to eliminate optical distortion and ensure epipolar alignment of the images. This process is critical to ensure the accuracy of disparity calculation and the reliability of depth estimation.

[0104] First, a dual-camera system with precise time synchronization is used. The two cameras are synchronized via a hardware trigger to ensure that images are captured within the same time window. The captured images are then distortion corrected based on the previous camera calibration data (including intrinsic parameters and distortion coefficients). A mathematical model is used to precisely adjust each pixel in the image to remove radial and tangential distortion caused by the characteristics of the camera lens. The distortion-corrected images are epipolar corrected to ensure that the projections of the same 3D spatial point in the left and right images are strictly aligned to the same horizontal line. The images are adjusted to achieve epipolar alignment by calculating and applying a stereo correction transformation (using the camera's extrinsic parameters).

[0105] Step 303: Multi-scale pyramid decomposition

[0106] Get the original image from the synchronized binocular camera system. Perform necessary preprocessing on the original image, including color correction and contrast adjustment, to ensure that the image quality is suitable for subsequent processing. In order to accurately cover the disparity calculation of different scales, a three-layer image pyramid is constructed. This number of layers provides different disparity resolutions from coarse to fine, which is suitable for depth estimation in various scenes. The specific layering method is as follows:

[0107] First layer (original layer): The first layer uses the processed original image. This layer is not downsampled and retains the highest resolution image to ensure that all detailed scene information is included.

[0108] Second layer: Gaussian blur is applied to the image in the first layer, smoothed with a Gaussian filter with a kernel size of 5×5, and then downsampled to reduce the resolution to 50% of the original image. The specific operation is to sample every other pixel, that is, to select half of the pixels in each row and column of the image.

[0109] Third layer (highest resolution): Gaussian blur is performed on the second layer again, using the same kernel size, and then another round of downsampling is performed, further reducing the image resolution to 50% of the second layer, which is 25% of the original resolution.

[0110] Step 304: Disparity calculation

[0111] Disparity calculation is performed independently at each level of the image pyramid. The semi-global block matching (SGBM) algorithm is a stereo matching algorithm based on global optimization, which considers the pixels of the entire image or most of it, and solves the disparity by minimizing the energy function in multiple directions. This method not only considers the brightness difference between pixels, but also introduces a smoothing term to ensure the continuity of the disparity, thereby effectively handling occlusion and texture discontinuity areas in the disparity calculation.

[0112] To estimate the disparity, we first need to calculate the matching cost. This step involves evaluating the similarity of each pixel in the left image to the corresponding pixel at the disparity position in the right image. The cost function measures the brightness difference:

[0113] C(p,d)=|I L (p)-I R (pd)|,

[0114] Where C(p,d) is the cost of pixel p under disparity d, I L and I R They are the left and right images respectively, and || means taking the absolute value.

[0115] The calculated matching costs are then aggregated in multiple directions to improve the accuracy and robustness of disparity estimation. The SGBM algorithm aggregates costs along multiple paths through dynamic programming. The aggregation process can be expressed as:

[0116] S(p,d)=C(p,d)+min(S(pr,d),S(pr,d-1)+P1,S(pr,d+1)+P1,S(pr,d)+P2),

[0117] Where S(p,d) is the aggregated cost, r is the considered path direction, and P1 and P2 are smoothness penalty parameters used to adjust the continuity of the disparity and the allowed disparity change.

[0118] Finally, the disparity value with the minimum cost is selected for each pixel from the aggregated cost, all possible disparity values ​​of each pixel are traversed, and the disparity value that minimizes the aggregated cost is selected as the final disparity of the pixel.

[0119] Step 305: Multi-scale parallax fusion

[0120] After calculating the disparity of all layers, the results of different layers are fused to produce the final high-precision disparity map. In multi-scale disparity fusion, correct weight allocation is crucial, which directly affects the quality of the fused disparity map. The main purpose of weight allocation is to utilize the different information provided by each image layer and reasonably allocate its influence on the final disparity map through scientific methods, thereby optimizing the overall performance of the disparity map.

[0121] First, the weight of each image layer is determined based on its resolution. Image layers with higher resolution can provide more detailed disparity information and therefore have an advantage in weight distribution. The weight of each image layer is proportional to its resolution. Specifically, the weight is defined by the product of the resolution of the image layer and its gradient strength:

[0122]

[0123] Among them, Resolution(i) represents the resolution ratio of the i-th layer image relative to the highest resolution layer, and Gradient Magnitude(I i ) represents the sum of the gradient amplitudes of the i-th layer image, and the maximum value is the sum of the maximum gradient amplitudes in all layers.

[0124] In the previous step, an initial weight was calculated based on the resolution of each image layer. This weight reflects the contribution of the image layer in an ideal situation, but in actual disparity calculation, due to various factors such as occlusion, lighting changes, or low-texture areas, the disparity results of some image layers may not be reliable enough. Therefore, the purpose of the second step is to adjust these initial weights based on the confidence of the disparity calculation of each image layer. The specific steps are to adjust the weight according to the disparity confidence of each pixel point, and the formula is:

[0125]

[0126] Among them, w i (p) is the local weight for pixel p, C min (p) and C second min (p) are the minimum and sub-minimum of the disparity cost, β is the normalization factor, and C threshold It is a preset threshold used to adjust the sensitivity of the weight.

[0127] In this way, the weight of each pixel now not only reflects the inherent characteristics of the image layer, but also takes into account the confidence of the actual disparity calculation. This comprehensive weight can better reflect the true contribution of each image layer to the quality of the final disparity map, thereby improving the accuracy and reliability of the fused disparity map.

[0128] Using the adjusted weights calculated in the first and second steps, the disparity data of each image layer is weightedly fused to synthesize a final disparity map that integrates all layers of data to provide higher accuracy and visual continuity. For each pixel p, the weighted average disparity value is calculated using the disparity data of all layers and the corresponding weights. The formula for weighted average is as follows:

[0129]

[0130] Where D i (p) is the disparity value of pixel p in the disparity map of the i-th layer, w i,p It is the weight calculated by combining resolution and disparity confidence.

[0131] Through the above method, it is ensured that the disparity information obtained from each image layer is reasonably and effectively used in the final disparity map, which improves the adaptability and effect of the multi-scale disparity fusion strategy. The detailed process is shown in the figure. This detailed fusion strategy is particularly suitable for processing disparity calculation tasks in industrial construction scenes.

[0132] Step 306: Calculate depth to achieve distance measurement

[0133] After completing the multi-scale disparity fusion, we get a high-quality disparity map that integrates the data of each image layer. The next step is to use this fused disparity map to calculate the actual tilt depth in three-dimensional space. In the stereo vision system, the depth Z of the object can be calculated by the following formula:

[0134]

[0135] Among them, f is the focal length of the camera, B is the baseline distance between the two cameras, and d is the disparity value, that is, the data in the fused disparity map.

[0136] For each pixel in the depth map, the above formula is applied to calculate its corresponding distance tilt depth, and the depth value of each pixel is converted into a distance value to generate a panoramic ranging map, which provides the actual distance from the camera to the object for each pixel.

[0137] A depth threshold δ is set. If the state predicted in step 2 and the calculated depth exceed the set threshold, a tower tilt warning is issued.

[0138] Embodiment 2

[0139] This embodiment provides an infrastructure site management and control system based on multi-source data fusion, including:

[0140] A multi-source data acquisition module, which is used to acquire the posture data of the transmission tower and the image data of the infrastructure site;

[0141] The tilt angle prediction module is used to obtain the predicted value of the tilt angle of the transmission tower according to the posture data of the transmission tower and the trained tower tilt angle prediction model; wherein the construction process of the tower tilt angle prediction model is: extracting the posture data sequence of the transmission tower with correlation from the posture data of the transmission tower; using the regression algorithm in machine learning, taking the posture data sequence of the transmission tower with correlation as the input of the tower tilt angle prediction model, and fitting to obtain the predicted value of the tilt angle of the transmission tower;

[0142] The early warning judgment module is used to combine the image data of the infrastructure site, use image layers of different resolutions to calculate and refine the disparity information layer by layer, and obtain a fused disparity map; based on the fused disparity map, the distance tilt depth corresponding to each pixel is calculated to obtain the actual tilt depth of the transmission tower in three-dimensional space; when the actual tilt depth is greater than the set depth threshold, a tower tilt warning is issued.

[0143] Embodiment 3

[0144] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the infrastructure site management and control method based on multi-source data fusion as described above are implemented.

[0145] Embodiment 4

[0146] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the infrastructure site management and control method based on multi-source data fusion as described above are implemented.

[0147] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0151] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A construction site control method based on multi-source data fusion, characterized in that: The steps include: Obtain the posture data of transmission towers and image data of infrastructure sites; The predicted value of the inclination angle of the transmission tower is obtained according to the posture data of the transmission tower and the trained tower inclination angle prediction model; wherein the construction process of the tower inclination angle prediction model is as follows: Extracting a transmission tower posture data sequence having correlation from the transmission tower posture data; The regression algorithm in machine learning is used to take the posture data sequence of the transmission tower with correlation as the input of the tower tilt angle prediction model, and the predicted value of the transmission tower tilt angle is obtained by fitting. Combined with the image data from the infrastructure site, the disparity information is calculated and refined layer by layer using image layers of different resolutions to obtain a fused disparity map. The distance tilt depth corresponding to each pixel is calculated based on the fused disparity map to obtain the actual tilt depth of the transmission tower in three-dimensional space. When the actual tilt depth is greater than the set depth threshold, a tower tilt warning is issued.

2. A construction site control method based on multi-source data fusion as claimed in claim 1, characterized in that: The posture data of the transmission tower includes the forced vibration status of the transmission tower, the inclination angle of the transmission tower, the meteorological data of the transmission tower operation site, the temperature of the transmission wire and cable joints, and the strain status data of the cement tower foundation.

3. The infrastructure site control method based on multi-source data fusion according to claim 1, characterized in that: A dynamic sliding window is used to extract a sequence of posture data of transmission towers with correlation in the posture data of transmission towers, and the posture data of transmission towers with correlation are used as training samples. The training samples include two parts: features and labels. The training samples are continuous historical posture data of transmission towers, and the labels are the posture data of transmission towers that follow the training features.

4. A construction site control method based on multi-source data fusion as claimed in claim 3, characterized in that: The length of the dynamic sliding window is: Among them, p i is the ratio of the i-th data to the sum of the data sequence, is the average value of the data sequence, n is the number of data sequences, {x1,x2…,x n } is the data sequence.

5. The infrastructure site control method based on multi-source data fusion according to claim 1, characterized in that: The tower tilt angle prediction model training specifically includes: Pre-training the tower inclination angle prediction model, during which a neural network including multiple restricted Boltzmann machines and a BP neural network is used; each restricted Boltzmann machine includes an explicit layer and a hidden layer, the bottom explicit layer is used as an input layer, the input layer and multiple hidden layers form a deep belief network for extracting features of sample data, and the last hidden layer and the output layer form a BP neural network; The overall parameters of the pre-trained tower inclination angle prediction model are reversely fine-tuned, wherein the reverse fine-tuning includes: obtaining the corresponding output according to the input of the pre-trained neural network; recalculating the weights and biases of the deep belief network based on the back-propagation algorithm; and modifying the weights and biases of the deep belief network based on the momentum gradient descent method.

6. The infrastructure site control method based on multi-source data fusion according to claim 1, characterized in that: The method of calculating and refining disparity information layer by layer based on the predicted value of the tilt angle of the transmission tower and the image data of the infrastructure site using image layers of different resolutions to obtain a fused disparity map includes: A three-layer image pyramid is constructed. The first layer retains the highest resolution image. The second layer performs Gaussian blur processing on the image of the first layer, and then downsamples it to reduce the resolution to 50% of the original image to obtain the second resolution image. The third layer performs Gaussian blur processing on the second layer again to further reduce the image resolution to 50% of the second layer to obtain the third resolution image. The disparity calculation is performed independently at each layer of the image pyramid. The calculation process includes: The matching cost is calculated based on the similarity between each pixel in the left image and the corresponding disparity position pixel in the right image; Aggregate costs along multiple paths through dynamic programming to obtain aggregated costs; Select the disparity value with the minimum cost for each pixel from the aggregated cost, traverse all possible disparity values ​​of each pixel, and select the disparity value that minimizes the aggregated cost as the final disparity of the pixel; After calculating the disparity of all layers, the results of different layers are fused to obtain a fused disparity map.

7. A construction site control method based on multi-source data fusion as claimed in claim 6, characterized in that: After calculating the disparity of all layers, the results of different layers are fused to obtain a fused disparity map, including: Determine the weight basis and resolution of each image layer, where the weight of each image layer is proportional to its resolution; Adjust the initial weights according to the confidence of the disparity calculation at each image layer; For each pixel p, the weighted average disparity value is calculated using the disparity data of all layers and the corresponding weights.

8. A construction site management and control system based on multi-source data fusion, characterized in that: include: A multi-source data acquisition module, which is used to acquire the posture data of the transmission tower and the image data of the infrastructure site; The tilt angle prediction module is used to obtain the predicted value of the tilt angle of the transmission tower according to the posture data of the transmission tower and the trained tower tilt angle prediction model; wherein the construction process of the tower tilt angle prediction model is: extracting the posture data sequence of the transmission tower with correlation from the posture data of the transmission tower; using the regression algorithm in machine learning, taking the posture data sequence of the transmission tower with correlation as the input of the tower tilt angle prediction model, and fitting to obtain the predicted value of the tilt angle of the transmission tower; The early warning judgment module is used to combine the image data of the infrastructure site, use image layers of different resolutions to calculate and refine the disparity information layer by layer, and obtain a fused disparity map; based on the fused disparity map, the distance tilt depth corresponding to each pixel is calculated to obtain the actual tilt depth of the transmission tower in three-dimensional space; when the actual tilt depth is greater than the set depth threshold, a tower tilt warning is issued.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a construction site management and control method based on multi-source data fusion as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the infrastructure site management and control method based on multi-source data fusion as described in any one of claims 1-7 are implemented.

Images