Monitoring system based on radar interferometry and photogrammetry
Patent Information
- Authority / Receiving Office
- AU · AU
- Patent Type
- Applications
- Current Assignee / Owner
- IDS GEORADAR SRL
- Filing Date
- 2025-02-19
- Publication Date
- 2026-07-30
AI Technical Summary
Existing radar interferometry systems lack intuitive visualization of data overlap with photogrammetry and struggle to provide high-precision phase measurements for deformation and vibration monitoring, particularly in scenarios requiring high spatial resolution and compact, versatile operation.
A system integrating a MIMO radar with stereo cameras, configured to generate virtual antennas and perform interferometry and stereometric photogrammetry, allowing for high-frequency radar image acquisition and automatic mode switching based on distance, enabling precise deformation and vibration monitoring with intuitive data overlap.
Enables rapid, accurate monitoring of both deformations and vibrations with intuitive visualization, supporting compact and versatile operation at various distances, and enhancing spatial resolution through high-frequency radar imaging and photogrammetric reconstruction.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Monitoring system based on radar interferometry and photogrammetry DESCRIPTIONField of the invention
[0001] The present invention relates to the field of structural monitoring by means of interferometry.
[0002] In particular, the invention relates to a system for monitoring the deformation and vibrations of a scenario that integrates interferometry technology with photogrammetry technology.Description of the prior art
[0003] As is well known, in order to monitor the stability of structures such as houses, bridges or structures of historical and cultural interest, monitoring devices based on interferometric radar technology are often used. This technology is already widely used for the critical monitoring of landslides in mining contexts. The purpose of these monitoring systems is to provide information regarding the deformations and vibrations of the structure, taking the appropriate measures in case of detection of anomalous movements (evacuation of buildings, structural restoration interventions) .
[0004] An interferometric radar system measures the amplitude and phase of the signals reflected from the monitored scenario. The amplitude provides information on the strength of the reflected signal, while the phase can be correlated to the relative movement of the target towards or away from the radar. The magnitude of the displacement is obtained with the interferometric technique, which relates the difference in phase measurement that occurs between a first and a second acquisition with the line-of- sight displacement of the monitored surface.
[0005] Radar interferometry provides displacement measurements with sub-millimeter precision and is therefore very effective in detecting deformation of the structure of interest. At the same time, this technology can operate without the need to install reflectors on the scenario (as is necessary in laser technology), exploiting only the natural reflectivity of the monitored surfaces.
[0006] However, an important limitation of this technology is that radar data is generally not very intuitive in its original form and it is not easy to associate the radar information with a specific object or area within the monitored scenario.
[0007] For this reason, systems have been developed that integrate interferometry technology with photogrammetrytechnology, using a radar system and an optical system comprising one or more stereo cameras.
[0008] An example is provided by the document "MIMO-SAR: A Hierarchical High-resolution Imaging Algorithm for mmWave FMCW Radar in Autonomous Driving" published by Xiangyu Gao et al., which describes a MIMO-SAR algorithm applied to a system that integrates a MIMO system and a stereo camera.
[0009] However, the system was designed for automotive target detection and not for deformation monitoring, which requires high spatial resolution and high accuracy in terms of phase measurement. Furthermore, stereo cameras only have a horizontal baseline and are positioned at the same height above the ground; this design reduces the photogrammetric capabilities for detecting horizontal elements in the scene. Also for this reason, no overlap is produced between the radar data and the camera image.
[0010] Another example is provided by the document "Muitifunctional and Compact 3D FMCW MIMO Radar System with Rectangular Array for Medium Range Applications" published by Enric Miralles et al., which describes a system consisting of a MIMO array capable of determining the position of a target in 3D with a single camera. The system is able to produce an overlay between the radar image and the camera image.
[0011] However, this system is aimed at detecting targets rather than monitoring deformations / vibrations, and therefore does not provide a high-precision phase measurement system.
[0012] Another example is provided by the paper "Tracking Deformation Processes at the Legnica Glogow Copper District (Poland) by Satellite InSAR—II: Zelazny Most Tailings Dam" published by Paolo Mazzanti et al., which describes a system in which deformation data based on satellite SAR are superimposed on optical images. However, the SAR resolution in this case is generated by the movement of the radar sensor, and not by a MIMO array. This type of radar regenerates the radar map very slowly (even over days), making it impossible to monitor the vibrations of the scenario.Summary of the invention
[0013] It is therefore a feature of the present invention to provide a system that allows monitoring of both deformations and vibrations of a scenario, through integration between interferometric technology and photogrammetric technology.
[0014] It is also a feature of the present invention to provide such a system that allows an intuitive visualization of the overlap between interferometric data and photogrammetric data.
[0015] It is still a feature of the present invention to provide such a system that is compact and versatile in monitoring scenarios located at different distances from the operator.
[0016] These and other objects are achieved by a system for monitoring a scenario comprising a device, said device comprising :- a frame integral to a reference system S(x,y,z);- a radar system comprising:— a number n ≥ 1 of transmitting antennas, each transmitting antenna having a geometric centerhaving coordinates with i= 1,2,...,n;— a number m ≥4 of receiving antennas, each receiving antenna having a geometric centerhaving coordinates with j= 1,2,...,m;said radar system generating an array of virtual antennas comprising a number p=n*m of virtual antennas, said array of virtual antennas defining a geometric center each virtual antenna havinga geometric center having coordinateswhere and- an optical system arranged to frame a field of view D of said scenario, said optical system comprising: a first camera having a geometric centerhaving coordinates— a second camera having a geometric centerhaving coordinatessaid geometric center of said first camera andsaid geometric center of said second cameradefining a geometric offset along said axis x equal to and a geometric offset along saidaxis z equal tosaid system furthermore comprising:- a display;- a control unit arranged to:— by means of said radar system (120), carry out interferometry of said scenario obtaining a displacement map of said scenario comprising a plurality of cells, wherein each cell of said displacement map is localized with respect to said reference system S(x,y,z);— by means of said optical system, carry out stereometric photogrammetry of said scenario obtaining a point cloud of said scenario comprising a plurality of points, wherein each point of said point cloud is localized with respect to said reference system S(x,y,z);— by means of said point cloud of said scenario, carry out an overlapping between said displacementmap and an image of said scenario acquired by said optical system, obtaining a composite map of said scenario comprising both said displacement map and said image of said scenario;— show on said display said composite map of said scenario; said radar system and said optical system both having pointing direction parallel to the axis y of said reference system S(x,y,z), said radar system being configured in such a way that at least one of the following geometric constraints is met:- Lx> λ;- Lz> λ; where Lxand Lzare the maximum distance between two geometric centres respectively, along the axis xand along the axis z of said reference system S(x,y,z), and λ is the central wavelength of the transmitted signal band, whose main feature is that said radar system is configured to acquire a number of radar images per second higher than 10, that said control unit is configured to switch between:- a single-shot mode, wherein said control unit is adapted to obtain said point cloud of said scenarioby the comparison between a first image, acquired by said first camera at a time t1, and a second image, acquired by said second camera at a time t2= t1;- a multi-shot mode, wherein said control unit is adapted to obtain said point cloud of said scenario by the comparison between at least one first image, acquired by said first camera and / or by said second camera at a time t1, and at least one second image, acquired by said first camera and / or by said second camera at a time t2> t1; and that said control unit is also arranged to:- require said user to select a portion of said field of view D of said scenario, said portion of said field of view D comprising a number q of pixels rk, with k = 1,2, q;- by means of said optical system, calculate a distance dkbetween said device and each pixel rkpresent in said portion of said field of view D of said scenario;- calculate an average distance dmbetween said device and said portion of said field of view D of said scenario, where- in case that said average distance dmis less than a threshold distance dt, automatically switch tosaid single-shot mode;- in case that said average distance dmis greater than said threshold distance dt, automatically switch to said multi-shot mode.
[0017] Thanks to the high acquisition frequency of the radar system and the arrangement of the virtual antennas, the present invention allows the monitoring not only of deformations but also of vibrations of the scenario, making it possible to extrapolate more information about the state of the scenario itself and, possibly, allowing for rapid intervention .
[0018] Furthermore, the present invention allows the user to use the single-shot mode when he is in front of a scenario located at a distance less than a threshold distance dtof, for example, 30 meters (i.e., in short-range mode), and to use instead the multi-shot mode when the scenario is located at a distance greater than this threshold distance dt(i.e. mode long-range mode).
[0019] In the first mode, the user can take a single shot with both cameras simultaneously and create the point cloud from the comparison of the two images, while in the second mode the user can take several shots in time sequence by framing the same area of the scene from different camera positions.
[0020] In this way, the system can automatically switch to the most appropriate mode of use, based on the distance between the portion of the scene in focus and the optical system.
[0021] In particular, the locution "portion of the field of view D of the scenario" refers to a percentage between 0% and 100% of this field of view D . Therefore, the user can also select the entire field of view D of the scenario.
[0022] In particular, said radar system is a MIMO radar system, i.e. a "Multiple-inlet multiple-output" radar system.
[0023] The MIMO radar system allows the generation of virtual antennas and therefore contributes to the monitoring of vibrations in the scenario.
[0024] Advantageously, both said geometric offset Bxand said geometric offset Bzare greater than zero.
[0025] The diagonal geometric offset between the camera centres allows for a photogrammetric 3D reconstruction of the scenario that is accurate whether the scenario includes mainly vertical elements, or mainly horizontal elements, or both. Essentially, this arrangement of the cameras allows a good reconstruction of the three-dimensional point cloud of any possible scenario.
[0026] In particular, Bx> 1cm and Bz>1cm.
[0027] Advantageously, in the event that the condition dm> dtis verified, said control unit is also arranged to suggest to said user to carry out a movement and to provide geometric information regarding the distance to be kept between each acquired image and the next in said multi-shot mode.
[0028] Advantageously, during said step of carrying out an overlapping between said displacement map and an image of said scenario acquired by said optical system, said control unit is adapted to:- associate each point of said point cloud to a relative cell of said displacement map, obtaining a displacement value dLOSof each point of said point cloud;- associate each point of said point cloud to a relative pixel of an image of said scenario acquired by said optical system;- carry out an overlapping between each cell of said displacement map and a relative pixel of said image of said scenario.
[0029] In particular, said control unit is also arranged to: for each point of said point cloud, calculate a direction normal to the surface of said scenario at said point; for each point of said point cloud, calculate a"line of sight" direction, which virtually connects said point of said point cloud with said geometric center of said array of virtual antennas;for each point of said point cloud, calculate an angle θ between said normal direction and said line of sight; for each point of said point cloud, calculate a displacement value dnof said point with respect to the normal direction by the equation dn= dLOS / cosθ;- generate a normalized displacement map wherein each displacement value of a respective cell is calculated as displacement value dn;- overlap said normalized displacement map with said image of said scenario acquired by said optical system, obtaining a normalized composite map.Brief description of the drawings
[0030] The invention will be now shown with the following description of its exemplary embodiments, made by way of example and not limited to, with reference to the attached drawings in which:- Fig. 1 shows a possible exemplary embodiment of the system for monitoring a scenario, according to the present invention;- Fig. 2 diagrammatically shows the geometry of an array of virtual antennas generated starting froma configuration, purely exemplary, of receiving antennas and transmitting antennas;- Fig. 3 diagrammatically shows the following steps performed by the control unit;- Fig. 4A shows a possible arrangement of a plurality of devices of the system for monitoring a scenario, according to the present invention, when the control unit is in the single-shot mode;- Fig. 4B shows a possible arrangement of the device of the system for monitoring a scenario, according to the present invention, and its progressive movement, when the control unit is in the multishot mode;- Figs. 5A and 5B diagrammatically show the movement of the user during a possible image acquisition session in the multi-shot mode.Description of some preferred exemplary embodiments
[0031] With reference to Fig. 1, a system for monitoring a scenario comprises a device 100 comprising a frame 110, integral with a reference system S(x,y,z), a radar system 120 and an optical system 130. In particular, the radar system 120 and the optical system 130 both have pointing direction parallel to the axis y of the reference system S(x,y,z).
[0032] The system also comprises a control unit and a display 140, each of which can be internal or external tothe device 100. For example, the display 140 can be located in a remote position with respect to the device 100 and be connected to it cable in wireless mode.
[0033] In particular, the radar system 120 comprises a number n ≥ 1 of transmitting antennas 121, each transmitting antenna 121 having a geometric center having coordinateswith i= 1,2,...,n, and a number m ≥4 of receivingantennas 122, each receiving antenna 121 having a geometric center having coordinates with j = 1,2,
[0034] With reference even to Fig. 2, the radar system 120 is adapted to generate a number p = n*m of virtual antennas, each virtual antenna having a geometric center havingcoordinates where
[0035] In the exemplary embodiment of Figs. 1 and 2, purely by way of example, the radar system 120 comprises 3 transmitting antennas 121 and 4 receiving antennas 122, thus generating 12 virtual antennas.
[0036] For example, the radar system 120 is a MIMO type radar system, i.e. "Multiple-inlet multiple-output".
[0037] In particular, furthermore, the optical system 120 is adapted to frame a field of view D of the scenario and comprises a first camera 131 having a geometric centerhaving coordinates and a second camera (132) havinga geometric center having coordinates
[0038] As shown in the figure, the geometric center ofthe first camera 131 and the geometric center of thesecond camera 132 define a geometric offset along the axis x equal to and a geometric offset along the axisz equal to In the present exemplary embodiment,Bx> 1cm and Bz> 1cm .
[0039] With reference also to the flow chart 300 of Fig.3, the control unit is adapted to carry out interferometry of the scenario by means of the radar system 120, obtaining a displacement map of the scenario comprising a plurality of cells, wherein each cell is localized with respect to the reference system S(x,y,z)
[0301] .
[0040] Subsequently, previously or in parallel, the control unit is adapted to carry out stereometric photogrammetry of the scenario by means of the optical system 130, obtaining a point cloud comprising a plurality of points, wherein each point is localized with respect to the reference system S(x,y,z)
[0302] .
[0041] Subsequently, the control unit, owing to the point cloud obtained, is adapted to associate each point of the point cloud to a relative cell of the displacement map
[0303] , associate each point of the point cloud to a relative pixel of an image of the scenario acquired by the optical system
[0304] and then carry out an overlap between the displacement map and the image of the scenario acquired bythe optical system, obtaining a composite map of the scenario
[0305] .
[0042] Such composite map of the scenario is then shown on the display 140
[0306] .
[0043] Advantageously, the radar system 120 is configured to acquire a number of radar images per second higher than 10 and the transmitting antennas 121 and the receiving antennas 122 are located geometrically in order to respect at least one of the following geometric constraints:where Lxand Lzare a maximum distance between two geometric centres respectively, along the axis x and along theaxis z of said reference system S(x,y,z), and λ is the central wavelength of the transmitted signal band.
[0044] In this way, thanks to the high acquisition frequency of the radar system 120 and the arrangement of the transmitting antennas 121 and receiving antennas 122, and therefore to the consequent arrangement of the virtual antennas, the present invention allows the monitoring not only of the deformations but also of the vibrations of the scenario, allowing to extrapolate more information regarding the state of the scenario itself and, possibly, allowing a rapid intervention.
[0045] With reference even to Figs. 4A and 4B, in a possible exemplary embodiment, the control unit is configured to pass between a single-shot mode (single-shot stereo acquisition) and a multi-shot mode (multi-shot acquisition).
[0046] In particular, in the single-shot mode the control unit is adapted to obtain the point cloud of the scenario by the comparison between a first image, acquired by the first camera 131 at a time t1, and a second image acquired by the second camera 132 at a time t2=.
[0047] In the multi-shot mode, instead, the control unit is adapted to obtain the point cloud of the scenario by the comparison between at least one first image, acquired by the first camera 131 and / or by the second camera 132 at a time t1, and at least one second image acquired by the first camera 131 and / or by the second camera 132 at a time t2>t1.
[0048] This function allows the user to use the singleshot mode when he is faced with a scenario located at a distance less than a threshold distance dt, for example, of 30 metres (i.e. short-range mode), and to use the multishot mode instead when the scenario is located at a distance greater than this threshold distance dt(i.e. long-range mode).
[0049] In the first mode, the user can take a single shot with both cameras simultaneously and create the point cloudby comparing the two images, while in the second mode the user can take several shots in a time sequence framing the same area of the scenario from different framing positions.
[0050] In a possible exemplary embodiment, the control unit is also arranged to:- require the user to select a portion of the field of view D of the scenario comprising a number q of pixels rk, with k = 1,2,...,q;- by means of the optical system 130, calculate a distance dkbetween the device 100 and each pixel rkpresent in the portion of the field of view D of the scenario;- calculate an average distance dmbetween the device 100 and the portion of the field of view D of the scenario, where dm=- in case that the average distance dmis less than a threshold distance dt, automatically switch to the single-shot mode;- in case that the average distance dmis greater than the threshold distance dt, automatically switch to the multi-shot mode.
[0051] In this way, the system can automatically switch to the most appropriate mode of use, based on the distance between the portion of the framed scenario and the optical system.
[0052] In particular, the locution "portion of the field of view D of the scenario" refers to a percentage between 0% and 100% of this field of view D . Therefore, the user can also select the entire field of view D of the scenario.
[0053] With reference even at Figs. 5A and 5B, in the event that the condition dm> dtis verified and therefore the system switches to the multi-shot mode, the control unit is adapted to suggest information to the user regarding the movement to be made between each acquired image and the next.
[0054] In particular, by defining as "baseline" the overall distance B traveled by the user during the multi-shot operation, i.e. the distance between the position of the first image and the position of the last image acquired, it is possible to define as quantity b the distance to be covered between the acquisition of one image and the next, in a direction substantially transversal to the direction of aiming of the cameras, obtaining the relation: b = (Nsh- 1) / B where Nsflis the number of the shots taken.
[0055] Such amount b can also be converted in number of steps Nstto be carried out using the relation:Nst= ceil(b / Δs) where As= 0,6m is the indicative length of a step.
[0056] The control unit is adapted to calculate an appropriate ratio B / dm, in order to minimize triangulation errors and therefore improve the quality of the photogrammetric reconstruction, and to suggest to the user the corresponding value of b or Nst.
[0057] The foregoing description exemplary embodiments of the invention will so fully reveal the invention according to the conceptual field of view, so that others, by applying current knowledge, will be able to modify and / or adapt for various applications such embodiment without further research and without parting from the invention, and, accordingly, it is therefore to be understood that such adaptations and modifications will have to be considered as equivalent to the specific embodiments. The means and the materials to realise the different functions described herein could have a different nature without, for this reason, departing from the field of the invention. It is to be understood that the phraseology or terminology that is employed herein is for the purpose of description and not of limitation.
Claims
CLAIMS1. A system for monitoring a scenario comprising a device (100), said device (100) comprising:- a frame (110) integral to a reference system S(x,y,z);- a radar system (120) comprising:— a number n ≥1 of transmitting antennas (121), each transmitting antenna (121) having a geometric center having coordinates with i=1,2,— a number m ≥4 of receiving antennas (122), each receiving antenna (121) having a geometric center having coordinates with j= 1,2,said radar system (120) generating an array of virtual antennas comprising a number p=n*m of virtual antennas, said array of virtual antennas defining a geometric center each virtualantenna having a geometric center havingcoordinates whereand- an optical system (130) arranged to frame a field of view D of said scenario, said optical system (130) comprising:— a first camera (131) having a geometric center having coordinates— a second camera (132) having a geometric center having coordinatessaid geometric center of said first camera (131)and said geometric center of said second camera(132) defining a geometric offset along said axis x equal to and a geometric offset alongsaid axis z equal tosaid system comprising furthermore:- a display (140);- a control unit arranged to:— by means of said radar system (120), carry out interferometry of said scenario obtaining a displacement map of said scenario comprising a plurality of cells, wherein each cell of said displacement map is localized with respect to said reference system S(x,y,z);— by means of said optical system (130), carry out stereometric photogrammetry of said scenario obtaining a point cloud of said scenario comprising a plurality of points, wherein each point of said point cloud is localized with respect to said reference system S(x,y,z);— by means of said point cloud of said scenario, carry out an overlap between said displacement map and an image of said scenario acquired by saidoptical system, obtaining a composite map of said scenario comprising both said displacement map and said image of said scenario;— show on said display said composite map of said scenario; said radar system (120) and said optical system (130) both having pointing direction parallel to the axis y of said reference system S(x,y,z), said radar system (120) being configured in such a way that at least one of the following geometric constraints is met:- Lx> λ;- Lz> λ; where Lxand Lzare the maximum distance between two geometric centres respectively, along the axis xand along the axis z of said reference system S(x,y,z), and λ is the central wavelength of the transmitted signal band, said system for monitoring a scenario characterized in that said radar system (120) is configured to acquire a number of radar images per second higher than 10, in that said control unit is configured to switch between:- a single-shot mode, wherein said control unit is adapted to obtain said point cloud of said scenarioby the comparison between a first image, acquired by said first camera (131) at a time t1, and a second image, acquired by said second camera (132) at a time t2= t1;- a multi-shot mode, wherein said control unit is adapted to obtain said point cloud of said scenario by the comparison between at least one first image, acquired by said first camera (131) and / or by said second camera (132) at a time t1, and at least one second image, acquired by said first camera (131) and / or by said second camera (132) at a time t2> t1; and in that said control unit is further arranged to:- require said user to select a portion of said field of view D of said scenario, said portion of said field of view D comprising a number q of pixels rk, with k = 1,2,...,q;- by means of said optical system (130), calculate a distance dkbetween said device (100) and each pixel rkpresent in said portion of said field of view D of said scenario;- calculate an average distance dmbetween said device (100) and said portion of said field of view D of said scenario, where dm=- in case that said average distance dmis less than a threshold distance dt, automatically switch tosaid single-shot mode;- in case that said average distance dmis greater than said threshold distance dt, automatically switch to said multi-shot mode.
2. The system for monitoring a scenario, according to claim 1, wherein said radar system (120) is a MIMO radar system, i.e. "Multiple-inlet multiple-output".
3. The system for monitoring a scenario, according to claim 1, wherein both said geometric offset Bxand said geometric offset Bzare greater than zero.
4. The system for monitoring a scenario, according to claim 3, wherein Bx> 1cm and Bz> 1cm.
5. The system for monitoring a scenario, according to claim 1, wherein, in the event that the condition dm> dtis verified, said control unit is also arranged to suggest to said user to carry out a movement and to provide geometric information regarding the distance to be kept between each acquired image and the next.
6. The system for monitoring a scenario, according to claim 1, wherein, during said step of carrying out an overlapping between said displacement map and an image of said scenario acquired by said optical system, said control unit is adapted to:- associate each point of said point cloud to arelative cell of said displacement map, obtaining a displacement value dLOSof each point of said point cloud;- associate each point of said point cloud to a relative pixel of an image of said scenario acquired by said optical system;- carry out an overlapping between each cell of said displacement map and a relative pixel of said image of said scenario.
7. The system for monitoring a scenario, according to claim6, wherein said control unit is also arranged to: for each point of said point cloud, calculate a direction normal to the surface of said scenario at said point; for each point of said point cloud, calculate a "line of sight" direction, which virtually connects said point of said point cloud with said geometric center of said array of virtual antennas;for each point of said point cloud, calculate an angle θ between said normal direction and said line of sight; for each point of said point cloud, calculate a displacement value dnof said point with respect to the normal direction by the equation dn= dLOS / cosθ;- generate a normalized displacement map wherein eachdisplacement value of a respective cell is calculated as displacement value dn;- overlap said normalized displacement map with said image of said scenario acquired by said optical system, obtaining a normalized composite map.