Building target monitoring method based on millimeter wave radar and video fusion technology

Abstract

The invention discloses a building target monitoring method based on a millimeter wave radar and a video fusion technology, and relates to the technical field of building monitoring. Comprising six steps, all parts in a video image are cut, separated and corrected by adopting a three-dimensional scene fusion mode, and then are fused into a three-dimensional scene, so that simple and efficient image fusion operation can be realized, fusion of a virtual scene and a real video is more accurate, and the building monitoring precision and efficiency are effectively improved. According to the building target monitoring method based on the millimeter-wave radar and the video fusion technology, rapid scanning of the building is achieved through nondestructive and non-contact of millimeter-wave near-field penetration imaging, a three-dimensional model of the building is formed, the whole area of the building can be monitored integrally, the monitoring data precision is high, and the monitoring efficiency is high. Long-distance, large-range and continuous monitoring can be achieved, and the problems that traditional deformation monitoring data are not reflected in time, and the monitoring data are modified manually are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of building monitoring, and specifically provides a building target monitoring method based on millimeter-wave radar and video fusion technology. Background Art

[0002] At present, building settlement, cracking, etc. are common deformation damage phenomena in buildings. The damages caused by common building deformations include settlement, cracking, inclination, and even the collapse of building (structure) structures, which affect normal use. Especially in the new construction stage of buildings, the settlement deformation is particularly obvious, and uneven settlement will bring about the inclination and tensile-compressive deformation of the building.

[0003] Therefore, building deformation monitoring is very important, and the monitoring of other aspects of the building state is also very important. However, in actual production and life, the monitoring technical means are single, and usually the laser monitoring method is used. This method requires marking targets on the monitored building, with a large labor intensity and unable to achieve continuous deformation monitoring of the building.

[0004] The invention with the publication number CN114612682A discloses a building state monitoring method and system based on video. First, a database of the building to be monitored is established and the information of the building to be monitored is stored; then, the video images of the building to be monitored are obtained, the monitoring target area is determined, and the features of this area are extracted. Finally, the monitoring result is obtained by comparing with the standard features in the database.

[0005] As shown in the above invention, the existing building monitoring methods generally collect images of the building and the building ground through video devices distributed around the building, and analyze and process them in combination with the building attribute information and geographical information to obtain the monitoring result of the building inclination state. However, this video method only simply superimposes the images in the overlapping area, seriously affecting the visual effect of the panoramic image, thereby reducing the accuracy of building state monitoring. Moreover, this method only monitors through the form of video data collection, with relatively low monitoring efficiency and poor monitoring accuracy. Summary of the Invention

[0006] Aiming at the deficiencies of the prior art, the present invention provides a building target monitoring method based on millimeter-wave radar and video fusion technology, which solves the existing problems.

[0007] To achieve the above objectives, the present invention is realized through the following technical solutions: A building target monitoring method based on millimeter-wave radar and video fusion technology, comprising the following steps:

[0008] Step 1: Deploy millimeter-wave radars in multiple directions of the target building, monitor the target building according to the millimeter-wave radars at different positions, obtain the monitoring data of the target building, and analyze the collected data to obtain the three-dimensional model of the target building;

[0009] Step 2: Obtain the remote sensing data image of the target building, and at the same time obtain multiple videos of the target building taken from different angles and the position information of the cameras;

[0010] Step 3: Preprocess the taken videos to obtain the preprocessed video data, and combine the remote sensing data image and the three-dimensional model of the target building to form the virtual scene of the target building;

[0011] Step 4: Calculate the monitoring parameters and graphic fusion matching parameters for each video monitoring point one by one according to the obtained video data. After cutting and separating the images in the video data according to the graphic fusion matching parameters, splice them in the virtual scene to obtain the video three-dimensional virtual scene fusion image;

[0012] Step 5: Extract the monitoring features of the target building, and query the corresponding image standard features in the building database according to the monitoring features of the target building;

[0013] Step 6: Calculate the feature deviation value of the building to be monitored according to the monitoring features and standard features, and judge the monitoring status of the building to be monitored according to the feature deviation value.

[0014] Preferably, in Step 1, the millimeter-wave radar equipment is used to complete the data collection of the building, and the near-field three-dimensional imaging technology is used to complete the imaging of the invisible parts to form the three-dimensional model of the target building.

[0015] Preferably, the preprocessing method in Step 3 is as follows:

[0016] The video data is decoded to obtain a single video image frame. A sample frame is extracted from each video stream, and the SIFT operator is used to find the feature point matching in the sample frame, and color consistency processing is performed.

[0017] Preferably, the color consistency processing is as follows:

[0018] 1) Extract a sample frame from each of the two videos for matching, construct the color histogram formed by all pixels in the frame, and through color histogram equalization and normalization processing, make the two video frames have the same color histogram distribution;

[0019] 2) Perform the same histogram equalization and normalization processing on each frame in the same video stream as the corresponding sample frame, thereby completing the consistency processing of the entire video stream;

[0020] 3) Create a cache for video frames, with a size capable of accommodating approximately 50 video frames (video frame resolution is 1920*1080);

[0021] 4) Load frame data using a first-in, first-out (FIFO) list structure.

[0022] Preferably, in step 4, according to the number of cameras and the video frame resolution, a panoramic image is established, and the overlapping area of two adjacent real-time video frames in the panoramic image is optimized by overlapping to obtain a panoramic image after overlapping optimization processing.

[0023] Preferably, the monitoring parameters in step 4 include the shooting alignment direction parameter and magnification of the camera, and the graphic fusion matching parameters include the position parameter extracted by image cutting from the video data, and the splicing parameter for projecting the cut and extracted image data in the virtual scene.

[0024] Preferably, the standard features of the target building in step 5 include the building design information data of the building, and the building design information data is obtained from the architectural design drawing or identified from the architectural design CAD drawing.

[0025] Preferably, the monitoring status of the building to be monitored in step 6 is judged according to the following steps:

[0026] Judge whether the feature deviation value exceeds a preset threshold;

[0027] If not, save the monitoring feature value and repeat the loop;

[0028] If so, send a monitoring result signal; and start a verification monitoring program to analyze and process the building to be monitored.

[0029] Advantageous Effects

[0030] The present invention provides a building target monitoring method based on millimeter-wave radar and video fusion technology. Compared with the prior art, it has the following advantageous effects:

[0031] 1. This building target monitoring method based on millimeter-wave radar and video fusion technology can cut, separate, and correct each part of the video image through a three-dimensional scene fusion method, and then integrate it into the three-dimensional scene, which can realize simple and efficient image fusion operations, make the fusion of the virtual scene and the real video more accurate, and effectively improve the monitoring accuracy and efficiency of the building.

[0032] 2. The building target monitoring method based on millimeter-wave radar and video fusion technology can quickly scan a building by means of the non-destructive and non-contact millimeter-wave near-field penetration imaging, form a three-dimensional model of the building, and can monitor an entire area of the building. Moreover, the monitoring data has high precision, enabling long-distance, large-scale, and continuous monitoring. It solves problems such as untimely reflection of traditional deformation monitoring data and artificial modification of monitoring data, and avoids losses of personnel and property caused by monitoring reasons. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the system flow of the present invention;

[0034] Figure 2 It is a schematic diagram of the video image and three-dimensional scene boundary calibration and image projection correction method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0036] Refer to Figure 1-2 , the present invention provides the following three technical solutions:

[0037] The first embodiment: A building target monitoring method based on millimeter-wave radar and video fusion technology includes the following steps:

[0038] Step 1: Deploy millimeter-wave radars at multiple orientations of the target building, monitor the target building according to the millimeter-wave radars at different positions, obtain the monitoring data of the target building, and analyze the collected data. Use the millimeter-wave radar equipment to complete the data collection of the building, and use the near-field three-dimensional imaging technology to complete the imaging of the invisible parts to form a three-dimensional model of the target building;

[0039] Step 2: Obtain the remote sensing data image of the target building, and at the same time obtain the videos of the target building taken from multiple different angles and the position information of the cameras;

[0040] Step 3: Preprocess the taken videos to obtain the preprocessed video data, and combine the remote sensing data image and the three-dimensional model of the target building to form a virtual scene of the target building;

[0041] Step 4: Establish a panoramic image according to the number of cameras and the video frame resolution. Optimize the overlapping area of two adjacent real-time video frames in the panoramic image to obtain the panoramic image data after overlapping optimization. Calculate the monitoring parameters and graphic fusion matching parameters for each video monitoring point based on the acquired video data one by one. After cutting and separating the images in the video data according to the graphic fusion matching parameters, splice them in the virtual scene to obtain a video three-dimensional virtual scene fusion image. The monitoring parameters include the shooting alignment direction parameter and the magnification of the camera, and the graphic fusion matching parameters include the position parameter extracted by image cutting from the video data and the splicing parameter for projecting the cut and extracted image data in the virtual scene;

[0042] Step 5: Extract the monitoring features of the target building, and query the corresponding image standard features in the building database according to the monitoring features of the target building. The standard features of the target building include the building design information data of the building, and the building design information data is obtained from the architectural design drawing or recognized from the architectural design CAD drawing;

[0043] Step 6: Calculate the feature deviation value of the building to be monitored according to the monitoring features and the standard features, and judge the monitoring status of the building to be monitored. The monitoring status of the building to be monitored is judged according to the following steps:

[0044] Judge whether the feature deviation value exceeds the preset threshold;

[0045] If not, save the monitoring feature value and repeat the loop;

[0046] If so, send a monitoring result signal; and start a verification monitoring program to analyze and process the building to be monitored.

[0047] The second implementation method: The method for preprocessing the video data of the target building in the above method is as follows:

[0048] Decode the video data to obtain a single video image frame. Extract a sample frame from each video stream, use the SIFT operator to find the feature point matching in the sample frame, and perform color consistency processing.

[0049] The color consistency processing is as follows:

[0050] 1) Extract a sample frame from each of the two videos for matching, construct a color histogram formed by all pixels in the frame, and through color histogram equalization and normalization processing, make the two video frames have the same color histogram distribution;

[0051] 2) Perform histogram equalization and specification processing on each frame in the same video stream in the same way as the corresponding sample frame, thereby completing the consistency processing for the entire video stream;

[0052] 3) Create a cache for video frames, with a size that can accommodate approximately 50 video frames (video frame resolution is 1920*1080);

[0053] 4) Load the frame data using a first-in, first-out (FIFO) list structure.

[0054] The third implementation mode: The steps of calculating the monitoring parameters and graphic fusion matching parameters for each of the video monitoring points one by one according to the video data include:

[0055] S1: Obtain the boundary points T: T1, T2... T of the device to be replaced in the scene of the three-dimensional model. n , and calculate the center point T of this boundary according to the boundary points T. c ; where n is greater than 3;

[0056] S2: Divide the area of the device to be replaced formed by surrounding the boundary points T into n triangles TR formed by any two consecutive boundary points T and the center point T. c , TR 12c , TR 23c ... TR (n-1)nc , TR n1c ; and decompose the three-dimensional model fragments corresponding to the triangle TR into corresponding three-dimensional model fragment sequences TRP: TRP 12c , TRP 23c ... TRP (n-1)nc , TRP n1c ;

[0057] S3: Obtain the boundary points S: S1, S2... S of the device to be replaced in the video data. n , and calculate the center point S of this boundary according to the boundary points S. c ;

[0058] S4: Divide the area of the device to be replaced formed by surrounding the boundary points S into n triangles SR formed by any two consecutive boundary points S and the center point S. c , SR 12c , SR 23c ... SR (n-1)nc , SR n1c ;

[0059] S5: Extract the image corresponding to the triangle SR from the video data, project it onto the triangle TR correspondingly, and cover it onto the corresponding three-dimensional model fragment sequence TRP12c , TRP 23c …TRP (n-1)nc , TRP n1c , in which the current boundary point T, center point T c , triangle TR, boundary point S, center point S c and triangle SR are recorded as the graphic fusion matching parameters.

[0060] Furthermore, S5 includes:

[0061] S51: Extract the image corresponding to triangle SR from the video data;

[0062] S52, Based on the triangular projection algorithm, project the extracted image corresponding to triangle SR into triangle TR;

[0063] S53, Cover the extracted image corresponding to triangle SR onto the corresponding three-dimensional model fragment sequence TRP 12c , TRP 23c ……TRP (n-1n、c , TRP n1c .

[0064] In this embodiment, after extracting the image corresponding to triangle SR from the video data, based on the triangular projection algorithm, project the extracted image corresponding to triangle SR into triangle TR; then cover the extracted image corresponding to triangle SR onto the corresponding three-dimensional model fragment sequence TRP. There will be corresponding deviations when the image is covered onto the corresponding three-dimensional model fragment sequence TRP, and projection correction is required.

[0065] The image projection correction method for correcting the projection is: Calculate the distance L (n-1)nc from any pixel SP i in SR (n-1) to S n , S (n+1) and S iS(n-1) , L iSn and L iS(n+1) . Let the coordinates of the projection point TP i corresponding to SP (n-1)nc in TRP i be (x, y), then it respectively corresponds to satisfying that the distance L (n-1) from this projection point to T n , T (n+1) and T iT(n-1) , L iTn and L iT(n+1) should satisfy the equation:

[0066] L iT(n-1) =K*L iS(n-1) ;

[0067] L iTn =K*L iSn ;

[0068] L iT(n+1) =K*L iS(n+1) ;

[0069] K is a fixed constant. The three equations are combined to solve the TP i Coordinates (x, y).

[0070] Specifically, refer to Figure 2 , with SR 12c The projection of is used as an example to illustrate: Calculate SR 12c Any pixel SP i Distance L to S1, S2 and S3 iS1 , L iS2 and L iS3 , let SP i In TRP 12c The corresponding projection point TP i The coordinates are (x, y), and the distance L from the projection point to T1, T2 and T3 should be satisfied respectively. iT1 , L iT2 and L iT3 Should meet:

[0071] L iT1 =K*L iS1 ;

[0072] L iT2 =K*L iS2 ;

[0073] L iT3 =K*L iS3 ;

[0074] Where K is a fixed constant that can be set by the management personnel. By combining the above three equations, the TPi coordinates x and y can be solved.

[0075] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art, and the model parameters of each electrical appliance are not specifically limited, and conventional equipment can be used.

[0076] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0077] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A building target monitoring method based on millimeter wave radar and video fusion technology, characterized in that: The following steps are involved: Step 1: Deploy millimeter-wave radars at multiple locations of the target building, monitor the target building according to the millimeter-wave radars at different locations, obtain monitoring data of the target building, and analyze the collected data to obtain a three-dimensional model of the target building; Step 2: Obtain remote sensing data images of the target building, and simultaneously obtain multiple videos of the target building shot at different angles and the location information of the camera; Step 3: Preprocess the captured video to obtain preprocessed video data, and form a virtual scene of the target building based on the remote sensing data image and the three-dimensional model of the target building; Step 4: Calculate the monitoring parameters and graphic fusion matching parameters for each video monitoring point one by one according to the acquired video data, cut and separate the images in the video data according to the graphic fusion matching parameters, and then splice them in the virtual scene to obtain a video three-dimensional virtual scene fusion image; Step 5: Extract the monitoring features of the target building, and query the building database for corresponding image standard features based on the monitoring features of the target building; Step 6: Calculate the characteristic deviation value of the building to be monitored based on the monitoring characteristics and standard characteristics, and determine the monitoring status of the building to be monitored based on the characteristic deviation value.

2. The building target monitoring method based on millimeter wave radar and video fusion technology according to claim 1 is characterized in that: In the step 1, the millimeter wave radar equipment is used to complete the data collection of the building, and the near-field three-dimensional imaging technology is used to complete the imaging of the invisible parts to form a three-dimensional model of the target building.

3. The building target monitoring method based on millimeter wave radar and video fusion technology according to claim 1 is characterized in that: The method of pretreatment in step 3 is as follows: The video data is decoded to obtain a single video image frame. A sample frame is extracted from each video stream, and the SIFT operator is used to find feature point matches in the sample frame, and color consistency processing is performed.

4. The building target monitoring method based on millimeter wave radar and video fusion technology according to claim 3 is characterized in that: The color consistency process is: 1) Extract a sample frame from each of the two videos to be matched, construct a color histogram formed by all pixels in the frame, and make the two video frames have the same color histogram distribution through color histogram equalization and regularization processing; 2) Perform the same histogram equalization and regularization processing on each frame in the same video stream as the corresponding sample frame, thereby completing the consistency processing of the entire video stream; 3) Create a buffer for video frames, the size of which can accommodate 50 video frames; 4) Use a first-in-first-out list structure to load frame data.

5. The building target monitoring method based on millimeter wave radar and video fusion technology according to claim 1 is characterized in that: In the step 4, a panoramic image is established according to the number of cameras and the video screen resolution, and overlapping optimization is performed on the overlapping area of ​​two adjacent real-time video screens in the panoramic image to obtain a panoramic image after overlapping optimization processing.

6. The building target monitoring method based on millimeter wave radar and video fusion technology according to claim 1 is characterized in that: The monitoring parameters in step 4 include the camera's shooting direction parameters and magnification, and the graphic fusion matching parameters include the position parameters for image cutting and extraction from video data, and the splicing parameters for projecting the cut and extracted image data in the virtual scene.

7. The building target monitoring method based on millimeter wave radar and video fusion technology according to claim 1 is characterized in that: The standard features of the target building in step 5 include building design information data of the building, and the building design information data is obtained from a building design drawing, or the building design information is identified from a building design CAD drawing.

8. The building target monitoring method based on millimeter wave radar and video fusion technology according to claim 1 is characterized in that: The monitoring status of the building to be monitored in step 6 is determined according to the following steps: Determine whether the characteristic deviation value exceeds a preset threshold; If not, the monitoring characteristic value is saved and the cycle is repeated; If yes, a monitoring result signal is sent; And initiate the verification and monitoring program to analyze and process the buildings to be monitored.

Images