Structural response monitoring method based on deep learning and fisheye camera

Through the combination of deep learning technology and fisheye cameras, the problem of geometric distortion in fisheye cameras is solved, and accurate monitoring of building structural displacement and health status evaluation is achieved, which extends the applicability of monitoring environment and improves the accuracy of monitoring data. Sex and reliability.

CN118195968BActive Publication Date: 2024-11-19HARBIN INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410333442.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-22
Publication Date
2024-11-19
Estimated Expiration
2044-03-22

AI Technical Summary

Technical Problem

The existing calculation methods do not consider the influence of geometric distortion in the fisheye camera, resulting in large errors in the structural response monitoring process.

Method used

The fisheye camera structure response monitoring method based on deep learning is adopted to eliminate geometric distortion through image distortion correction and feature extraction, and use wireframe extraction models and distortion correction models to eliminate geometric distortion, obtain displacement data of the building structure, and evaluate the health status of the building.

Benefits of technology

It realizes accurate monitoring of building structural displacement, expands the applicability of the monitoring environment, improves the accuracy and reliability of monitoring data, and is suitable for wide-angle shooting and local observation occasions, significantly improving task execution efficiency and result quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118195968B_ABST
    Figure CN118195968B_ABST
Patent Text Reader

Abstract

The present invention discloses a structural response monitoring method based on deep learning and fisheye camera, which belongs to the field of building structure monitoring. The method comprises the following steps: S100, using a fisheye camera to obtain raw data and preprocessing the data; S200, performing image distortion correction and feature extraction on the preprocessed data; S300, monitoring the displacement of the building structure based on the result output by S200. The present invention improves segmentation accuracy, enhances robustness, effectively utilizes prior information, and improves the consistency and interpretability of segmentation results. The structural response monitoring method of the present invention can be used for ordinary cameras and fisheye cameras, and has wider applicability. The technology of the present invention also includes a wireframe extraction model and a distortion correction model for ordinary cameras and fisheye cameras. These two models can also be widely used in other computer vision tasks such as image segmentation and image classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a structural response monitoring method based on deep learning and a fisheye camera, and belongs to the field of building structure monitoring. Background Art

[0002] By analyzing the collected structural response information, we can not only have an overall understanding of the condition of the structural system, but also make a proper assessment of the health and integrity of the structure. The traditional monitoring method of structural response is mainly to install a large number of acceleration / displacement sensors on the structure, which undoubtedly causes very expensive costs. In recent years, with the continuous development of computer vision methods, the use of traditional cameras to replace acceleration / displacement sensors has received widespread attention. As an advanced technical means, computer vision provides a new perspective and solution for structural response prediction. First, the use of cameras makes it possible to monitor building structures, equipment and systems in real time. Secondly, through comprehensive observation of the internal and external environment of the building, potential problems and anomalies can be discovered in time, providing an effective early warning mechanism for building maintenance. However, the current application of this method is only applicable to traditional cameras, and there is currently no better solution for the large number of fisheye cameras in cities.

[0003] Compared with traditional cameras, fisheye cameras can provide a viewing angle lens close to or equal to 180°, with a very large field of view observation range, and are an extreme wide-angle camera. Fisheye cameras can achieve large-scale, no-dead-angle monitoring with a small number of them, and can provide a large field of view while having high economic value. With the development of key projects such as my country's "Skynet Project" and "Safe City", the number of fisheye cameras currently accounts for a large proportion of urban monitoring cameras. Traditional cameras usually conform to the law of perspective projection to simulate the scenes seen by the human eye, while fisheye cameras use a nonlinear mapping method to obtain a wider field of view, but this method will cause serious geometric distortion in the generated images. The existence of the distortion effect currently brings difficulties and challenges to the fields of target detection and image recognition in computer vision tasks. Therefore, monitoring the required building structures based on fisheye cameras has a broad application prospect. Summary of the invention

[0004] The present invention aims to solve the problem that the existing calculation method does not take into account the influence of geometric distortion in the fisheye camera, which leads to large errors in the process of monitoring the structural response, and thus proposes a fisheye camera structural response prediction method based on deep learning. This method mainly considers the influence of geometric distortion, and extracts the "distortion line" in the picture to eliminate the error caused by geometric distortion through the "curve to straight" operation, so as to monitor the building structure response according to the results captured by the fisheye camera, solving the problems existing in the prior art.

[0005] The structural response monitoring method based on deep learning and fisheye camera includes the following steps:

[0006] S100, using a fisheye camera to obtain raw data and pre-process the data;

[0007] S200, performing image distortion correction and feature extraction on the preprocessed data;

[0008] S300: Perform displacement monitoring of the building structure based on the result outputted in S200.

[0009] Furthermore, in S100, the following steps are included:

[0010] S110, using a fisheye camera to obtain a video including displacement of a building structure;

[0011] S120, segmenting the video containing the displacement of the building structure frame by frame to obtain a series of images, namely, distorted original images.

[0012] S130, inputting the distorted original images into the wireframe extraction model one by one to obtain a distorted wireframe heat map of each image, wherein the distorted wireframe heat map depicts the distorted wireframe in the distorted original image, that is, the outline of the building.

[0013] Furthermore, in S130, the wireframe extraction model is constructed using a deep learning method, and the wireframe extraction model includes distorted images and normal images, which are used to obtain wireframe heat maps of different types of images.

[0014] Furthermore, in S200, the following steps are included:

[0015] S210, inputting the distortion line heat map into the distortion correction model, correcting the local curve features, and thereby determining the distortion parameters;

[0016] S220, applying the distortion parameters to the distorted original image to obtain a fully corrected image, i.e., a distortion-corrected image;

[0017] S230: input the distortion correction image into the wireframe extraction model again to obtain a corrected wireframe heat map.

[0018] Furthermore, in S210, the distortion correction model is constructed using a deep learning method, and the distortion correction model includes a local correction part based on the distortion line heat map and a global correction part based on the distorted original image.

[0019] Furthermore, the local correction part based on the distortion line heat map and the global correction part based on the distortion original image regress the distortion parameters respectively, and the corresponding distortion correction image is obtained after the regression data is processed.

[0020] Furthermore, in S300, specifically, the straight-line frame heat map and the distortion correction map are input into the structural response monitoring model, so as to obtain the displacement data of the building structure, and the health status of the building is evaluated according to the displacement data of the building structure.

[0021] Furthermore, the structural response monitoring model is constructed by deep learning method and optical flow method. Specifically:

[0022] Firstly, the wireframe extraction model is used to obtain the straight frame heat map of the distortion correction image. Then, the optical flow method is used to obtain the structural response based on the distortion correction image and its straight frame heat map. Finally, the final structural displacement time history curve is obtained through data fusion and Kalman filter.

[0023] A storage medium stores a computer program, which, when executed by a processor, implements the structural response monitoring method based on deep learning and a fisheye camera.

[0024] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the above-mentioned structural response monitoring method based on deep learning and a fisheye camera.

[0025] Beneficial effects of the present invention: The structural response monitoring method based on deep learning and fisheye camera of the present invention aims to accurately monitor the structural displacement of buildings through computer vision and deep learning technology:

[0026] (1) Compared with the prior art, the monitoring method not only supports traditional ordinary cameras, but is also compatible with fisheye cameras, greatly expanding the breadth of usage scenarios. This compatibility enables the method to be applied in more types of monitoring environments, whether it is a situation where wide-angle shooting is required to cover a larger area, or a situation where only an ordinary camera is needed for local observation, the monitoring method of the present invention can be adapted to provide stable and reliable monitoring performance.

[0027] (2) The wireframe extraction model and distortion correction model introduced in the present invention are specially designed for ordinary cameras and fisheye cameras. These two models not only play a key role in the structural displacement monitoring of the present invention, but also ensure the accuracy of the monitoring data by accurately extracting the building outline wireframe and effectively correcting the distortion; they can also be widely used in other computer vision tasks, such as image segmentation, image classification, etc. By deploying these models in different visual processing tasks, the execution efficiency and result quality of the tasks can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the structural response monitoring method based on deep learning and fisheye camera of the present invention;

[0029] Figure 2 It is a wireframe extraction model process framework diagram of the present invention;

[0030] Figure 3 This is a flow chart of the distortion correction model of the present invention. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] Reference Figure 1 As shown, the structural response monitoring method based on deep learning and fisheye camera includes the following steps:

[0033] S100, using a fisheye camera to obtain raw data and pre-process the data;

[0034] S200, performing image distortion correction and feature extraction on the preprocessed data;

[0035] S300: Perform displacement monitoring of the building structure based on the result outputted in S200.

[0036] Specifically, a structural response monitoring method based on deep learning and fisheye cameras is used for real-time monitoring and safety assessment of building structures. By using a fisheye camera to capture wide-angle, high-resolution raw data, and combining a deep learning algorithm to perform data preprocessing, image distortion correction, and feature extraction, this method can accurately monitor the displacement changes of building structures. The main advantage of the present invention is that it can provide wide coverage and high-precision monitoring. At the same time, the application of deep learning algorithms makes data processing more efficient and accurate, greatly improving the reliability and practicality of structural monitoring. This is of great significance for ensuring building safety, preventing potential risks, and responding to emergencies in a timely manner.

[0037] Furthermore, in S100, the following steps are included:

[0038] S110, using a fisheye camera to obtain a video including displacement of a building structure;

[0039] S120, segmenting the video containing the displacement of the building structure frame by frame to obtain a series of images, namely, distorted original images.

[0040] S130, inputting the distorted original images into the wireframe extraction model one by one to obtain a distorted wireframe heat map of each image, wherein the distorted wireframe heat map depicts the distorted wireframe in the distorted original image, that is, the outline of the building.

[0041] Specifically, in the S100 stage, first, a fisheye camera is used to obtain a video containing the displacement of the building structure, which can capture a wide range of viewing angles and more detailed information, thereby improving the comprehensiveness of monitoring. Subsequently, by segmenting the video frame by frame into a series of pictures, namely the distorted original pictures, this step makes the subsequent processing more flexible and efficient. Most importantly, these distorted original pictures are input into the wireframe extraction model one by one, and the generated distorted wire heat map not only intuitively depicts the outline of the building, but also highlights the distortion caused by the structural displacement, greatly improving the accuracy and intuitiveness of displacement monitoring.

[0042] Reference Figure 2 As shown, Figure 2 This is a flowchart of the wireframe extraction model process. In the figure, Conv3×3 is a convolution operation with a 3×3 convolution kernel, Batchnorm2d is batch normalization, Maxpooling is a maximum pooling operation, ConvTranspose2d is a deconvolution operation, Conv1×1 is a convolution operation with a 1×1 convolution kernel, and Relu and Sigmoid represent two different activation functions. “+” represents data addition and fusion.

[0043] Furthermore, in S130, the wireframe extraction model is constructed using a deep learning method, and the wireframe extraction model includes distorted images and normal images, which are used to obtain wireframe heat maps of different types of images.

[0044] Specifically, in the S130 stage, the wireframe extraction model constructed by the deep learning method is used to achieve efficient and accurate extraction of wireframes in distorted and normal images of buildings. This method can automatically identify and generate wireframe heat maps of different types of images, thereby accurately depicting the structural outline of the building and possible structural displacement. The introduction of deep learning has greatly improved the generalization ability and recognition accuracy of the model, making the wireframe extraction process more intelligent and automated. This not only reduces the need for manual intervention, but also improves processing speed and efficiency, ensuring the reliability and real-time nature of the monitoring results.

[0045] Furthermore, in S200, the following steps are included:

[0046] S210, inputting the distortion line heat map into the distortion correction model, correcting the local curve features, and thereby determining the distortion parameters;

[0047] S220, applying the distortion parameters to the distorted original image to obtain a fully corrected image, i.e., a distortion-corrected image;

[0048] S230: input the distortion correction image into the wireframe extraction model again to obtain a corrected wireframe heat map.

[0049] Specifically, in the S200 stage, the correction of image distortion and the optimization of wireframe extraction are effectively achieved. First, by inputting the distorted line heat map into the distortion correction model (S210), the technical solution can accurately correct the local curve features in the image, thereby accurately determining the distortion parameters. Subsequently, in step S220, these distortion parameters are applied to the distorted original image to achieve comprehensive correction of the image and obtain a clear and accurate distortion correction map. Finally, by inputting the distortion correction map into the wireframe extraction model (S230), the corrected straight line frame heat map is further obtained.

[0050] This step of S200 not only improves the accuracy and efficiency of distorted image correction, but also ensures the accuracy and reliability of subsequent wireframe extraction. By accurately correcting distortion, the solution can restore the true structure and details of the image, thereby providing a more accurate wireframe heat map. This highly automated and intelligent processing flow significantly improves the quality and speed of image processing, providing strong technical support for application fields such as building monitoring, urban planning, and map drawing, and has important practical application value and broad application prospects.

[0051] Reference Figure 3 As shown, Figure 3 This is the framework diagram of the distortion correction model process. In the figure, Flatten is the flattening operation, and Unflatten is the unflattening operation. FC is the fully connected layer. The rest of the process is the same as Figure 2The meaning is the same as in.

[0052] Furthermore, in S210, the distortion correction model is constructed using a deep learning method, and the distortion correction model includes a local correction part based on the distortion line heat map and a global correction part based on the distorted original image.

[0053] Specifically, in the S210 stage, the distortion correction model constructed by the deep learning method achieves efficient and accurate image distortion correction. The model uniquely combines the local correction part based on the distortion line heat map with the global correction part based on the distorted original image, so that the correction process not only focuses on the fine adjustment of local details, but also optimizes the image quality at the global level. The advantages of this method include: first, it can more accurately identify and correct the distortion problems in the image, whether it is a tiny local curve or an overall shape distortion; second, it ensures that the corrected image maintains a high degree of accuracy and naturalness in both details and the whole. This deep learning-driven comprehensive correction strategy greatly improves the effect of distortion correction and provides a more reliable and accurate basis for subsequent image analysis and application. Such a processing method not only significantly improves the processing efficiency and accuracy, but also enhances the applicability and flexibility of the model.

[0054] Furthermore, the local correction part based on the distortion line heat map and the global correction part based on the distortion original image regress the distortion parameters respectively, and the corresponding distortion correction image is obtained after the regression data is processed.

[0055] Specifically, through advanced processing flow, the local correction part based on the distortion line heat map and the global correction part based on the distorted original image are used to perform precise regression analysis on the distortion parameters, and then the data of these two parts are fused to finally obtain the distortion correction map. The significant advantage of this method is that it combines the advantages of both local and global correction methods. It can not only carefully adjust and correct the local distortion of the image, ensure the high accuracy and naturalness of the details, but also optimize the overall visual effect of the image at the global level. This fusion processing method significantly improves the accuracy and efficiency of distortion correction, ensures the quality of the corrected image, and greatly enhances the use value and application scope of the image.

[0056] Furthermore, in S300, specifically, the straight-line frame heat map and the distortion correction map are input into the structural response monitoring model, so as to obtain the displacement data of the building structure, and the health status of the building is evaluated according to the displacement data of the building structure.

[0057] Specifically, by simultaneously inputting the straight-line frame heat map and the distortion correction map into the structural response monitoring model, the accurate displacement data of the building structure can be obtained, thereby effectively evaluating the health status of the building, which has significant practical value and advantages. This method uses advanced image processing technology to capture and analyze subtle changes in the building structure in the actual environment, ensuring the high accuracy and reliability of the data. Through the precise monitoring and analysis of the displacement of the building structure, the potential risks and damage of the structure can be discovered in time, providing a scientific basis for the maintenance and repair of the building. This not only helps to extend the service life of the building and improve the safety of the building, but also reduces maintenance costs and enhances the economic value of the building. Especially for the health status monitoring of those buildings located in earthquake zones, storm-prone areas or old buildings, this technical solution shows great application potential and broad market prospects.

[0058] Furthermore, the structural response monitoring model is constructed by deep learning method and optical flow method. Specifically:

[0059] Firstly, the wireframe extraction model is used to obtain the straight frame heat map of the distortion correction image. Then, the optical flow method is used to obtain the structural response based on the distortion correction image and its straight frame heat map. Finally, the final structural displacement time history curve is obtained through data fusion and Kalman filter.

[0060] Specifically, this method combines deep learning and the structural response monitoring model constructed by the optical flow method, and realizes the accurate monitoring and analysis of the displacement of the building structure through advanced processing technologies such as wireframe extraction, data fusion and Kalman filtering. First, the straight frame heat map of the distortion correction map is obtained through the wireframe extraction model, and then the optical flow method is used to accurately calculate the response of the structure based on the distortion correction map and its straight frame heat map. Finally, the data fusion and Kalman filter technology are used for fine processing to obtain the time history curve of the structural displacement. The application of this series of technologies not only greatly improves the accuracy and reliability of the displacement data, but also can monitor and evaluate the health status of the building structure in real time, which is of great significance for preventing potential structural safety problems. In addition, the efficiency and automation level of this solution have brought new technological breakthroughs in the field of building health monitoring, effectively reduced labor costs, improved the efficiency and effectiveness of monitoring work, and provided strong technical support for the long-term safety management and maintenance of buildings.

[0061] A storage medium stores a computer program, which, when executed by a processor, implements the structural response monitoring method based on deep learning and a fisheye camera.

[0062] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the above-mentioned structural response monitoring method based on deep learning and a fisheye camera.

[0063] Specifically, the present invention implements an efficient structural response monitoring method based on deep learning and fisheye camera by embedding a specific computer program into a storage medium and executing the program using a processor of a computer device. This solution effectively integrates storage media, computer devices (including memory and processors) and advanced computer programs, thereby making the structural response monitoring process automated and intelligent, with high accuracy and real-time performance. This method can use the wide-angle images captured by the fisheye camera for in-depth analysis, and accurately calculate the response changes of the structure through deep learning technology, providing a new solution for building health monitoring.

[0064] The application of this technology not only improves the accuracy and reliability of monitoring data, but also greatly improves the monitoring scope and efficiency, especially when conducting comprehensive monitoring of buildings and other structures. At the same time, the solution has a high degree of automation, which reduces labor costs and potential human errors, and ensures the continuity and stability of the monitoring process. More importantly, by real-time monitoring and analysis of the response of the building structure, potential problems and risks of the structure can be discovered in a timely manner, providing reliable data support for timely maintenance and reinforcement, thereby protecting people's lives and property. In general, this technical solution provides an efficient and intelligent solution tool for the field of building health monitoring, which helps to improve the safety and reliability of buildings.

[0065] The following is a specific implementation example of the present invention:

[0066] Step 1: Collect the public data sets of indoor and outdoor pictures of buildings at this stage, and process the collected data sets. Generate the input data and labels of the wireframe extraction model and fisheye correction model by randomly generating 4 distortion parameters. At the same time, generate a virtual data set when the building is displaced through Blender software to train the structural response monitoring model;

[0067] Step 2: Use deep learning methods to build a wireframe extraction model. The wireframe extraction model includes distorted images and normal images. The wireframe heatmaps of different types of images can be obtained through the wireframe extraction model;

[0068] Step 3: Use deep learning methods to build a distortion correction model. The distortion correction model is divided into two parts: local correction based on the distortion line heat map and global correction based on the distortion original image. These two parts regress the distortion parameters respectively, and obtain the corresponding distortion correction map after fusion processing of the data;

[0069] Step 4: Use deep learning methods and optical flow methods to build a structural response monitoring model. First, use the wireframe extraction model to obtain the straight frame heat map of the distortion correction map, then use the optical flow method to obtain the structural response based on the distortion correction map and its straight frame heat map, and finally obtain the final structural displacement time history curve through data fusion and Kalman filter;

[0070] Step 5: obtain a video containing the displacement of the building structure shot by a fisheye camera, decompose the video frame by frame into corresponding pictures, input the obtained distorted original pictures into the wireframe extraction model one by one, and obtain a distorted line heat map of each distorted original picture;

[0071] Step 6: The distortion correction model includes two parts: local wireframe feature correction and overall texture correction;

[0072] The distortion line heat map contains the local features of the distorted original image, so first input the distortion line heat map into the distortion correction model, straighten the corresponding curve, and map the corresponding distortion parameters. Then input the distorted original image into the distortion correction model, and map the corresponding distortion parameters on the basis of ensuring the image texture features. Then the two data are fused to finally obtain the distortion correction map;

[0073] Step 7: Input the obtained distortion correction image into the existing wireframe extraction model to obtain the corrected wireframe heat map;

[0074] Step 8: Input the obtained straight line frame heat map and distortion correction map into the structural response monitoring model respectively to obtain the monitoring data of the corresponding building structure.

[0075] Compared with the prior art, the structural response monitoring method of the present invention can be used for ordinary cameras and fisheye cameras, and has wider applicability. The technology of the present invention also includes a wireframe extraction model and a distortion correction model for ordinary cameras and fisheye cameras. These two models can also be widely used in other computer vision tasks such as image segmentation and image classification.

[0076] The structural response monitoring method based on deep learning and fisheye camera provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A structural response monitoring method based on deep learning and fisheye camera, characterized in that: The structural response monitoring method of deep learning and fisheye camera includes the following steps: S100, using a fisheye camera to obtain raw data and pre-process the data; S200, performing image distortion correction and feature extraction on the preprocessed data; S300, performing displacement monitoring of the building structure based on the result output by S200; In S100, the following steps are included: S110, using a fisheye camera to obtain a video including displacement of a building structure; S120, segmenting the video containing the displacement of the building structure frame by frame to obtain a series of pictures, namely, distorted original pictures; S130, inputting the distorted original images into the wireframe extraction model one by one to obtain a distorted wireframe heat map of each image, wherein the distorted wireframe heat map depicts the distorted wireframe in the distorted original image, that is, the outline of the building; In S130, the wireframe extraction model is constructed using a deep learning method, and the wireframe extraction model includes a distorted image and a normal image, which are used to obtain wireframe heat maps of different types of images; In S200, the following steps are included: S210, inputting the distortion line heat map into the distortion correction model, correcting the local curve features, and thereby determining the distortion parameters; S220, applying the distortion parameters to the distorted original image to obtain a fully corrected image, that is, a distortion-corrected image; S230, inputting the distortion correction image into the wireframe extraction model again to obtain a corrected straight line frame heat map; In S210, the distortion correction model is constructed by using a deep learning method, and the distortion correction model includes a local correction part based on the distortion line heat map and a global correction part based on the distortion original image; In S210, the local correction part based on the distortion line heat map and the global correction part based on the distortion original image respectively regress the distortion parameters, and the regressed data is processed to obtain a corresponding distortion correction image; In S300, specifically, the straight frame heat map and the distortion correction map are input into a structural response monitoring model, so as to obtain displacement data of the building structure, and the health status of the building is evaluated according to the displacement data of the building structure; The structural response monitoring model is constructed by deep learning method and optical flow method, specifically: Firstly, the wireframe extraction model is used to obtain the straight frame heat map of the distortion correction image. Then, the optical flow method is used to obtain the structural response based on the distortion correction image and its straight frame heat map. Finally, the final structural displacement time history curve is obtained through data fusion and Kalman filter.

2. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the structural response monitoring method based on deep learning and fisheye camera described in claim 1 is implemented.

3. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the structural response monitoring method based on deep learning and a fisheye camera as described in claim 1.