Method and system for matching and selecting measurement points of stereo vision and microwave thermal imaging

Abstract

The present invention provides a method and system for matching and selecting measurement points using stereo vision and microwave thermal imaging, including step S1: collecting information within the field of view and determining a reference target; step S2: constructing a coordinate system and establishing a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system; and step S3: matching and selecting targets or measurement points. This method overcomes the challenges of prior art microwave distance-angle thermal imaging measurement point selection, such as its lack of intuitiveness, high reliance on prior knowledge, and difficulty in accurately matching with structural feature points. By utilizing stereo vision to locate targets or measurement points within the field of view in three dimensions, the method enables measurement point selection based on visual pixel coordinates and accurate mapping to corresponding microwave distance-angle thermal imaging measurement points.

Description

Technical Field

[0001] The present invention relates to the technical field of microwave thermal imaging measurement point identification, matching and selection, and in particular to a method and system for stereo vision and microwave thermal imaging measurement point matching and selection. Background Art

[0002] Microwave sensing technology, with its advantages of strong penetration, high precision, anti-interference capabilities, and strong environmental adaptability, plays an important role in military, communications, radar, and other fields. Its ability to operate stably in complex environments provides effective solutions for precise measurement and covert sensing, making it an indispensable key technology in modern science and technology.

[0003] Microwave transceivers primarily rely on the reflection of electromagnetic waves when sensing targets. Microwave-based micro-motion sensing requires accurate measurement point location and selection. Using a multi-transmitter, multi-receiver linear frequency modulated continuous wave (LFM) microwave transceiver, multiple targets can be distinguished across different distance dimensions using beat frequencies. Furthermore, angle resolution can be achieved using the time difference between the same target and different receiving antennas, enabling full-field joint range-angle localization of targets or measurement points. Measurement point identification and recognition relies on the microwave transceiver's range-angle thermal imaging. Measurement point selection from the thermal image relies heavily on the tester's prior knowledge of the specific spatial distribution of the measured target or measurement point. When the test scenario is complex, the measurement points are irregularly arranged and numerous, or the test distance is long, measurement point selection is inefficient and inaccurate, or even unsuccessful. This hinders the widespread application of microwave-based full-field vibration and deformation displacement measurement technologies in practical engineering applications. Therefore, there is an urgent need for intuitive and easy-to-use microwave imaging thermal image measurement point identification, location, and selection techniques.

[0004] Chinese patent publication number CN217029797U discloses a vibration-damping microwave transceiver comprising a mounting base, a vibration-damping assembly, a microwave transceiver body, a limiting post, a heat sink, and a connection port. The bottom of the vibration-damping assembly is fixedly connected to the mounting base, and the top of the vibration-damping assembly is fixedly connected to the microwave transceiver body. Extending upward from the top of the mounting base are limit blocks located on the left and right sides of the microwave transceiver body. The limit blocks have limiting openings, and the limiting posts extend through the limiting openings and are threadedly connected to the microwave transceiver body. The heat sink is mounted on the top of the microwave transceiver body. The front surface of the microwave transceiver body is provided with a connection port. A control host is disposed within the microwave transceiver body, and the connection port is electrically connected to the control host. This patent document provides a microwave transceiver with good vibration-damping performance and is easy to install and fix. The present invention addresses the problem that the microwave transceiver's distance-angle heat map is not intuitive and difficult to match with structural feature points, which is fundamentally different from the technical problem addressed by this patent document. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for matching and selecting measurement points of stereo vision and microwave thermal imaging.

[0006] A method for selecting and matching stereo vision and microwave thermal imaging measurement points provided by the present invention includes:

[0007] Step S1: Collect information within the field of view and determine the reference target;

[0008] Step S2: constructing a coordinate system and establishing a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system;

[0009] Step S3: Match and select targets or measuring points.

[0010] Preferably, step S1 includes fixing the microwave transceiver and the stereo camera, selecting more than or equal to three non-collinear targets or measuring points that exist in the field of view of the stereo camera and the microwave transceiver, or placing calibration reference objects as reference targets.

[0011] Preferably, the calibration reference object includes greater than or equal to three non-collinear targets that can be simultaneously perceived by the microwave transceiver and the stereo camera. The calibration and matching processing module completes the matching and joint calibration of the targets in the stereo visual imaging and the microwave range-angle heat map, and generates a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

[0012] Preferably, the microwave transceiver includes transmitting and receiving electromagnetic wave signals and outputting baseband signals; the stereo camera includes a binocular camera, a structured light camera and a TOF camera, and the stereo camera uses stereo vision to output relevant information; the relevant information includes information including the parallax or depth of the target.

[0013] Preferably, step S2 includes the following sub-steps:

[0014] Step S2.1: Complete the reference target in the camera coordinate system OX c Y c Z c Positioning under the camera; establish the camera coordinate system OX with the optical center of a RGB camera of the stereo camera as the origin c Y c Z c ;

[0015] Among them, X c OY c The plane is parallel to the imaging plane of the RGB camera, Z c The positive direction is from the optical center of the RGB camera to the outside;

[0016] The depth information of the reference target is obtained as the Z coordinate based on the depth measurement principle of stereo vision, which includes the binocular parallax principle and the light pulse time-of-flight 3D imaging method. The X and Y coordinates of the reference target are calculated based on other principles, which include the pinhole imaging principle.

[0017] Step S2.2: Complete the reference target in the microwave transceiver antenna coordinate system OX r Y r Z r Positioning under; let the antenna plane of microwave transceiver be Y r OZ r flat;

[0018] Among them, the arrangement direction of the antenna array is Y r Axis direction, when looking at the antenna plane from the front, Y r The positive axis points to the right, Z r The positive axis points upward, X r The positive axis points out of the antenna plane, and the microwave transceiver antenna coordinate system OX r Y r Z r The origin is the position of the first channel;

[0019] For a target (R, θ) in the microwave imaging range-angle heat map, determine the Z coordinate of the measuring point in the microwave transceiver antenna coordinate system as z r , then the X and Y coordinates corresponding to the measuring point are:

[0020]

[0021] Where R is the distance value of the target in the microwave distance-angle heat map, θ is the angle value of the target in the microwave distance-angle heat map, and y r is the Y coordinate of the target in the microwave transceiver antenna coordinate system, x r is the X coordinate of the target in the microwave transceiver antenna coordinate system;

[0022] Step S2.3: Determine the three-dimensional coordinates of three or more reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as:

[0023]

[0024] Where n≥3, P cam is the three-dimensional coordinate of the reference target in the camera coordinate system; P radar is the three-dimensional coordinate of the reference target in the microwave transceiver antenna coordinate system;

[0025] Obtain the rotation matrix R and translation matrix T from the camera coordinate system to the microwave transceiver antenna coordinate system.

[0026] Preferably, the step S3 includes identifying and selecting the target or measuring point to be measured in the pixel coordinate system and completing the conversion to the microwave distance-angle thermal map coordinate system; selecting a target or measuring point to be measured (u tgt ,v tgt ), where u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt is the pixel value of the target or measuring point in the vertical direction; and uniquely determines the Z coordinate value of the target or measuring point in the camera coordinate system as z c_tgt , and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt ); its three-dimensional coordinates in the microwave transceiver antenna coordinate system (x r_tgt ,y r_tgt ,z r_tgt ):

[0027]

[0028] Where R is the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system, and T is the translation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system;

[0029] The three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ) into a microwave distance-angle heat map:

[0030]

[0031] Among them, R tgt is the distance value of the target or measuring point in the microwave distance-angle thermal map; θ tgt It is the angle value of the target or measuring point in the microwave distance-angle heat map.

[0032] Preferably, the step S3 further includes the conversion of the microwave distance-angle heat map coordinates to pixel coordinates, based on the rotation matrix R' and translation matrix T' of the measuring point from the microwave transceiver antenna coordinate system to the camera coordinate system, assuming that the coordinates of the target to be measured in the microwave distance-angle heat map are (R tgt ,θ tgt ), determine the Z coordinate value z of the target or measuring point in the microwave transceiver antenna coordinate system r_tgt , then the X and Y coordinates of the target or measurement point in the microwave transceiver antenna coordinate system are:

[0033]

[0034] Among them, R tgt is the distance value of the target in the microwave distance-angle heat map, θ tgt is the angle value of the target in the microwave distance-angle heat map, y r_tgt is the Y coordinate of the target or measurement point in the microwave transceiver antenna coordinate system; r_tgt is the X coordinate of the target or measurement point in the microwave transceiver antenna coordinate system;

[0035] Assume that the coordinates of the target or measuring point in the camera coordinate system are (x c_tgt ,y c_tgt ,z c_tgt ), then:

[0036]

[0037] Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are (u cpoint ,v cpoint ), the coordinates of the target or measuring point in the pixel coordinate system are (u tgt ,v tgt ),have:

[0038]

[0039] Among them, u cpoint v is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, cpoint is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point to be converted in the horizontal direction, v tgt is the pixel value of the target or measurement point to be converted in the vertical direction, and f is the focal length of the RGB camera.

[0040] According to the present invention, a system for matching and selecting measurement points of stereo vision and microwave thermal imaging is provided, comprising:

[0041] Module M1: collects information within the field of view and determines the reference target;

[0042] Module M2: Construct a coordinate system and establish a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system;

[0043] Module M3: Matching and selecting targets or measuring points.

[0044] Preferably, the module M1 includes a fixed microwave transceiver and a stereo camera, and selects more than or equal to three non-collinear targets or measurement points that exist in the field of view of the stereo camera and the microwave transceiver, or places calibration reference objects as reference targets.

[0045] Preferably, the calibration reference object includes greater than or equal to three non-collinear targets that can be simultaneously perceived by the microwave transceiver and the stereo camera. The calibration and matching processing module completes the matching and joint calibration of the targets in the stereo visual imaging and the microwave range-angle heat map, and generates a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

[0046] Preferably, the microwave transceiver includes transmitting and receiving electromagnetic wave signals and outputting baseband signals; the stereo camera includes a binocular camera, a structured light camera and a TOF camera, and the stereo camera uses stereo vision to output relevant information; the relevant information includes information including the parallax or depth of the target.

[0047] Preferably, the module M2 includes the following submodules:

[0048] Module M2.1: Complete the reference target in the camera coordinate system OX c Y c Z c Positioning under the camera; establish the camera coordinate system OX with the optical center of a RGB camera of the stereo camera as the origin c Y c Z c ;

[0049] Among them, X c OY c The plane is parallel to the imaging plane of the RGB camera, Z c The positive direction is from the optical center of the RGB camera to the outside;

[0050] The depth information of the reference target is obtained as the Z coordinate based on the depth measurement principle of stereo vision, which includes the binocular parallax principle and the light pulse time-of-flight 3D imaging method. The X and Y coordinates of the reference target are calculated based on other principles, which include the pinhole imaging principle.

[0051] Module M2.2: Complete the reference target in the microwave transceiver antenna coordinate system OX r Y r Z r Positioning under; let the antenna plane of microwave transceiver be Y r OZ r flat;

[0052] Among them, the arrangement direction of the antenna array is Y r Axis direction, when looking at the antenna plane from the front, Yr The positive axis points to the right, Z r The positive axis points upward, X r The positive axis points out of the antenna plane, and the microwave transceiver antenna coordinate system OX r Y r Z r The origin is the position of the first channel;

[0053] For a target (R, θ) in the microwave imaging range-angle heat map, determine the Z coordinate of the measuring point in the microwave transceiver antenna coordinate system as z r , then the X and Y coordinates corresponding to the measuring point are:

[0054]

[0055] Where R is the distance value of the target in the microwave distance-angle heat map, θ is the angle value of the target in the microwave distance-angle heat map, and y r is the Y coordinate of the target in the microwave transceiver antenna coordinate system, x r is the X coordinate of the target in the microwave transceiver antenna coordinate system;

[0056] Module M2.3: Determine the three-dimensional coordinates of three or more reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as:

[0057]

[0058] Where n≥3, P cam is the three-dimensional coordinate of the reference target in the camera coordinate system; P radar is the three-dimensional coordinate of the reference target in the microwave transceiver antenna coordinate system;

[0059] Obtain the rotation matrix R and translation matrix T from the camera coordinate system to the microwave transceiver antenna coordinate system.

[0060] Preferably, the module M3 includes identifying and selecting the target or measuring point to be measured in the pixel coordinate system and completing the conversion to the microwave distance-angle thermal map coordinate system; selecting a target or measuring point to be measured (u tgt ,v tgt ), where u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt is the pixel value of the target or measuring point in the vertical direction; and uniquely determines the Z coordinate value of the target or measuring point in the camera coordinate system as z c_tgt , and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt); its three-dimensional coordinates in the microwave transceiver antenna coordinate system (x r_tgt ,y r_tgt ,z r_tgt ):

[0061]

[0062] Where R is the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system, and T is the translation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system;

[0063] The three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ) into a microwave distance-angle heat map:

[0064]

[0065] Among them, R tgt is the distance value of the target or measuring point in the microwave distance-angle thermal map; θ tgt It is the angle value of the target or measuring point in the microwave distance-angle heat map.

[0066] Preferably, the module M3 further includes a conversion of microwave distance-angle heat map coordinates to pixel coordinates, based on the rotation matrix R' and translation matrix T' of the measuring point from the microwave transceiver antenna coordinate system to the camera coordinate system, assuming that the coordinates of the target to be measured in the microwave distance-angle heat map are (R tgt ,θ tgt ), determine the Z coordinate value z of the target or measuring point in the microwave transceiver antenna coordinate system r_tgt , then the X and Y coordinates of the target or measurement point in the microwave transceiver antenna coordinate system are:

[0067]

[0068] Among them, R tgt is the distance value of the target in the microwave distance-angle heat map, θ tgt is the angle value of the target in the microwave distance-angle heat map, y r_tgt is the Y coordinate of the target or measurement point in the microwave transceiver antenna coordinate system; r_tgt is the X coordinate of the target or measurement point in the microwave transceiver antenna coordinate system;

[0069] Assume that the coordinates of the target or measuring point in the camera coordinate system are (x c_tgt ,y c_tgt ,z c_tgt ), then:

[0070]

[0071] Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are (u cpoint ,v cpoint ), the coordinates of the target or measuring point in the pixel coordinate system are (u tgt ,v tgt ),have:

[0072]

[0073] Among them, u cpoint v is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, cpoint is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point to be converted in the horizontal direction, v tgt is the pixel value of the target or measurement point to be converted in the vertical direction, and f is the focal length of the RGB camera.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. The present invention overcomes the difficulties of the existing microwave distance-angle heat map measurement point selection, such as the non-intuitiveness, high dependence on prior knowledge, and difficulty in accurately matching with structural feature points. It uses stereo vision to perform three-dimensional spatial positioning of targets or measurement points within the field of view, and realizes the function of selecting measurement points from visual pixel coordinates and accurately mapping them to corresponding microwave distance-angle heat map measurement points.

[0076] 2. The present invention improves the convenience and efficiency of microwave distance-angle thermal map measurement point selection, realizes the measurement point matching and selection of the fusion of stereo vision and microwave thermal map imaging, and provides a technical means for the selection of microwave distance-angle thermal map measurement point coordinates and structural coordinate matching in complex scenes.

[0077] Other beneficial effects of the present invention will be explained through the introduction of specific technical features and technical solutions in the specific implementation methods. Those skilled in the art should be able to understand the beneficial technical effects brought about by the introduction of these technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0079] Figure 1 Flow chart of the method of the present invention.

[0080] Figure 2Schematic diagram of stereoscopic vision imaging and microwave transceiver heat map measurement point matching in an embodiment of the present invention.

[0081] Figure 3 Schematic diagram of the process of establishing a camera coordinate system in an embodiment of the present invention.

[0082] Figure 4 Schematic diagram of the process of establishing a microwave transceiver antenna coordinate system in an embodiment of the present invention.

[0083] Figure 5 This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0084] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0085] Existing microwave thermal mapping measurement point identification and selection relies on the tester's prior knowledge of the specific spatial distribution of the target structure or measurement points, as well as the principles of microwave perception. When the test scene is complex, the measurement points are irregularly arranged, and the number of measurement points is large, the efficiency and accuracy of measurement point identification and selection will be greatly reduced. The relatively abstract thermal map measurement points are often difficult to match with structural feature points. To address this problem, the target needs to be measured; and to collect measurement data, the target measurement points need to be located first. How to locate the target measurement points? By using a microwave transceiver and a stereo camera, and constructing a camera coordinate system and a microwave transceiver antenna coordinate system respectively, the three-dimensional coordinates of the target measurement points are obtained. At this point, although different measurement points can be intuitively expressed, the coordinates of the target measurement points cannot be converted between different coordinate systems. Therefore, it is necessary to calculate the coordinate transformation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system based on these coordinate points to complete the calibration process; however, it is still impossible to measure the target measuring point at this time; therefore, a pixel coordinate system is first established, the target or measuring point to be measured is selected in the RGB camera pixel plane, the three-dimensional coordinates in the camera coordinate system are determined, and the coordinate transformation matrix is ​​used to convert it to the microwave transceiver antenna coordinate system, and then converted to the microwave distance-angle heat map. Through calibration and measurement, the identification and selection of the target or measuring point to be measured in the pixel plane and the matching conversion of pixel coordinates to microwave distance-angle heat map are completed; for the convenience of calculation, the present invention can also realize the conversion of microwave distance-angle heat map coordinates to pixel coordinates.

[0086] Reference Figure 1 and Figure 2As shown in the figure, first, the microwave transceiver and stereo camera are fixed, and three or more non-collinear targets that exist in the field of view of the stereo camera and the microwave transceiver are selected, or calibration reference objects are placed as reference targets, and the corresponding three-dimensional coordinate point pairs of these reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system are determined. Then, the coordinate transformation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system is calculated based on these coordinate point pairs to complete the calibration process; then, the target or measuring point to be measured is selected in the RGB camera pixel plane, and its three-dimensional coordinates in the camera coordinate system are determined. With the help of the coordinate transformation matrix obtained in the calibration process, its coordinates are transformed into the microwave transceiver antenna coordinate system, and then further transformed into the microwave range-angle heat map to complete the matching and selection of the target or measuring point.

[0087] The above is a basic embodiment of the present invention. The technical solution of the present invention is further described below through a preferred embodiment.

[0088] Example 1

[0089] Reference Figure 1 As shown, a method for matching and selecting measurement points using stereo vision and microwave thermal imaging includes:

[0090] Step 1: Calibration of the coordinate transformation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system:

[0091] First, set up the calibration scene:

[0092] Fix the microwave transceiver and stereo camera so that they remain relatively stationary and their fields of view cover the area where the target or measurement point is located. Select at least three non-collinear targets within the field of view that are visible in both the microwave transceiver and stereo camera images, or use calibration reference objects as reference targets.

[0093] Secondly, the reference target is in the camera coordinate system OX c Y c Z c Positioning below.

[0094] Reference Figure 3 As shown, the camera coordinate system OX is established with the optical center of one of the RGB cameras of the stereo camera as the origin c Y c Z c , where X c OY c The plane is parallel to the imaging plane of the RGB camera, Z cThe positive direction points outward from the optical center of the RGB camera. Stereoscopic vision, including binocular parallax and light pulse time-of-flight methods, is used to obtain the depth information of the reference target, namely the Z coordinate of the reference target in the aforementioned camera coordinate system. Furthermore, methods including pinhole imaging are used to calculate the X and Y coordinates of the reference target in the aforementioned coordinate system.

[0095] In this embodiment, two cameras with parallel optical axes, consistent focal lengths and located in the same plane are used to form a binocular camera. For a certain target or measuring point, the imaging coordinates on the left camera pixel plane and the right camera pixel plane are (u l ,v l ) and (u r ,v r ), the coordinates of the left camera and the right camera in the image plane coordinate system are (x l ,y l ) and (x r ,y r ), establish the camera coordinate system OX with the optical center of the left camera as the origin c Y c Z c , then the coordinates of the target or measuring point in the camera coordinate system (x c ,y c ,z c )for:

[0096]

[0097] Where b is the baseline distance of the binocular cameras, and f is the focal length of the two cameras.

[0098] Then the reference target is located in the microwave transceiver antenna coordinate system OX r Y r Z r Positioning below.

[0099] Reference Figure 4 As shown, let the antenna plane of the microwave transceiver be Y r OZ r plane, where the arrangement direction of the linear antenna array is Y r When looking at the antenna plane from the front, Y r The positive axis points to the right, Z r The positive axis points upward, X r The positive axis points outside the antenna plane, and the antenna coordinate system is OX r Y r Z r The origin is the position of the first virtual channel, that is, the midpoint of the line connecting the first transmitting antenna and the first receiving antenna.

[0100] For a target (R, θ) in the microwave imaging range-angle heat map, the Z coordinate of the measuring point in the microwave transceiver antenna coordinate system is determined to be z r , then the X and Y coordinates corresponding to the measuring point are:

[0101]

[0102] Where R is the distance value of the target in the microwave distance-angle heat map, θ is the angle value of the target in the microwave distance-angle heat map, and y r is the Y coordinate of the target in the microwave transceiver antenna coordinate system, x r is the X coordinate of the target in the microwave transceiver antenna coordinate system. r Negative values ​​are discarded because the front of the microwave transceiver antenna plane is X r Axis positive, the rear of the microwave transceiver antenna plane is X r In the negative direction of the axis, the target can be sensed by the microwave transceiver only if it is in front of the antenna plane.

[0103] The mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system is established by means of coordinate transformation matrix:

[0104] Determine the three-dimensional coordinates of more than or equal to three reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as:

[0105]

[0106] Where n≥3, P cam are the three-dimensional coordinates of these targets in the camera coordinate system; P radar are the three-dimensional coordinates of these targets in the microwave transceiver antenna coordinate system.

[0107] The rotation matrix R and translation matrix T from the camera coordinate system to the microwave transceiver antenna coordinate system are obtained by the singular value decomposition method. The specific process is:

[0108] Let the average coordinates of these targets in the camera coordinate system be The average coordinate in the microwave transceiver antenna coordinate system is Right now:

[0109]

[0110] Constructing a Matrix in Indicates P cam The i-th column of Indicates P radar The i-th column of . Perform singular value decomposition of matrix M:

[0111] M=UΣV T ;

[0112] Then the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system is:

[0113] R=VU T ;

[0114] The translation matrix is:

[0115]

[0116] Step 2: Identify and select the target or measurement point in the pixel coordinate system and complete the conversion to the microwave distance-angle thermal map coordinates.

[0117] Select a target or measurement point (u tgt ,v tgt ), where u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt The pixel value of the target or measuring point in the vertical direction; by referring to the target in the camera coordinate system OX c Y c Z c The positioning under the camera coordinate system uniquely determines its depth information, that is, the Z coordinate value z of the target or measuring point in the camera coordinate system c_tgt , and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt Then, the obtained rotation matrix R and translation matrix T are used to obtain its three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ).

[0118] For a target or measurement point (u l_tgt ,v l_tgt ), its coordinates (x l_tgt ,y l_tgt ) and (x r_tgt ,y r_tgt ), and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt Then, the obtained rotation matrix R and translation matrix T are used to obtain its three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ):

[0119]

[0120] Finally, the three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ) into a microwave distance-angle heat map:

[0121]

[0122] Among them, R tgt is the distance value of the target in the microwave distance-angle heat map, θ tgt is the angle value of the target in the microwave distance-angle heat map, thus completing the target or measurement point in the left camera pixel plane coordinate (u l_tgt ,v l_tgt ) to the microwave distance-angle heat map coordinates (R tgt ,θ tgt ) mapping and conversion.

[0123] In addition, the present invention can also realize the conversion of microwave distance-angle heat map coordinates to pixel coordinates. The specific method is as follows:

[0124] Still using the three-dimensional coordinates of the reference target obtained above in the microwave transceiver antenna coordinate system and the camera coordinate system, construct the matrix Calculate the rotation matrix R' and translation matrix T' of the measurement point from the microwave transceiver antenna coordinate system to the camera coordinate system.

[0125] Assume that the coordinates of the target to be measured in the microwave distance-angle thermal map are (R tgt ,θ tgt ), through measurement or prior knowledge, determine the Z coordinate of the target or measurement point in the microwave transceiver antenna coordinate system as z r_tgt , then the X and Y coordinates of the target or measurement point in the microwave transceiver antenna coordinate system are:

[0126]

[0127] Among them, y r_tgt is the Y coordinate of the target or measurement point in the microwave transceiver antenna coordinate system; r_tgt The X coordinate of the target or measurement point in the microwave transceiver antenna coordinate system.

[0128] Assume that the coordinates of the target or measuring point in the camera coordinate system are (x c_tgt ,y c_tgt ,z c_tgt ), then:

[0129]

[0130] Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are (u cpoint ,v cpoint ), the coordinates of the target or measuring point in the pixel coordinate system are (u tgt ,v tgt ),have:

[0131]

[0132] Among them, u cpoint v is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, cpoint is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point to be converted in the horizontal direction, v tgt is the pixel value of the target or measurement point to be converted in the vertical direction, and f is the focal length of the RGB camera.

[0133] This completes the measurement of the target or measuring point in the microwave distance-angle thermal map coordinates (R tgt ,θ tgt ) to RGB camera pixel plane coordinates (u tgt ,v tgt ) mapping and conversion.

[0134] The present invention overcomes the difficulties of the prior art in microwave distance-angle heat map measurement point selection, such as the lack of intuition, high dependence on prior knowledge, and difficulty in accurate matching with structural feature points. It uses stereo vision to perform three-dimensional spatial positioning of targets or measurement points in the field of view, realizes the function of selecting measurement points from visual pixel coordinates and accurately mapping them to corresponding microwave distance-angle heat map measurement points, improves the convenience and efficiency of microwave distance-angle heat map measurement point selection, realizes measurement point matching and selection that integrates stereo vision and microwave heat map imaging, and provides a technical means for microwave distance-angle heat map measurement point coordinate selection and structural coordinate matching in complex scenes.

[0135] Reference Figure 5 As shown, the present invention also provides a system for matching and selecting measurement points of stereo vision and microwave thermal imaging, comprising:

[0136] Microwave transceiver, stereo camera, calibration reference, signal acquisition module, calibration and matching processing module and display and storage module.

[0137] Among them, the microwave transceiver is used to transmit and receive electromagnetic wave signals, output baseband signals, and realize distance-angle thermal map positioning and imaging.

[0138] Stereo camera: Uses stereo vision modalities including binocular cameras, structured light cameras, and TOF cameras to output information including the parallax or depth of the target.

[0139] Calibration reference objects: include more than or equal to three non-collinear targets that can be perceived simultaneously by the microwave transceiver and the stereo camera, and are used for joint calibration and matching of targets in stereo vision imaging and microwave range-angle thermal maps.

[0140] Signal acquisition module: used to collect baseband signals from microwave transceivers, as well as raw images and disparity or depth information from stereo cameras.

[0141] The calibration and matching processing module includes the following processing steps: processing the baseband signal output by the microwave transceiver, obtaining the distance-angle heat map, and converting the coordinates of the target or measurement point to the microwave transceiver antenna coordinate system; locating the target in three dimensions in the camera coordinate system based on the parallax or depth information output by the stereo vision module; calibrating and generating the coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system; and calculating the corresponding microwave distance-angle heat map coordinate values ​​obtained by conversion based on the coordinate transformation matrix and the pixel coordinates selected in the RGB camera pixel coordinate system.

[0142] The display and storage module processes the following: displaying and outputting stereo camera visual imaging, depth maps, and microwave range-angle thermal map information for user calibration and measurement point selection and matching; displaying the coordinates of selected pixel points and synchronously displaying the corresponding microwave range-angle thermal map coordinates; and storing relevant coordinate information and visual image, depth map, and microwave range-angle thermal map imaging information.

[0143] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0144] In the description of this application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0145] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for matching and selecting measurement points of stereo vision and microwave thermal imaging, characterized in that: include: Step S1: Collect information within the field of view and determine the reference target; Step S2: constructing a coordinate system and establishing a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system; Step S3: matching and selecting targets or measuring points; The step S2 includes the following sub-steps: Step S2.1: Complete the reference target in the camera coordinate system Positioning under the camera; establish the camera coordinate system with the optical center of a RGB camera of the stereo camera as the origin ; in, The plane is parallel to the imaging plane of the RGB camera, The positive direction is from the optical center of the RGB camera to the outside; Based on the depth measurement principle of stereo vision, the depth information of the reference target is obtained as the Z coordinate; the stereo vision includes the binocular parallax principle and the light pulse flight time method 3D imaging; based on the pinhole imaging principle, the reference target is calculated X Coordinates and Y coordinate; Step S2.2: Complete the reference target in the microwave transceiver antenna coordinate system Positioning under; Assume that the antenna plane of the microwave transceiver is flat; Among them, the arrangement direction of the antenna array is Axis direction, when looking at the antenna plane from the front, The positive axis points to the right, The positive axis points upwards, The positive axis points out of the antenna plane, the microwave transceiver antenna coordinate system The origin is the position of the first channel; For a target in the microwave imaging range-angle heat map , determine the target or measurement point to be measured in the microwave transceiver antenna coordinate system Z The coordinates are , then the target or measuring point corresponds to X 、 Y The coordinates are: ； in, is the distance value of the target or measuring point in the microwave distance-angle thermal map, is the angle value of the target or measuring point in the microwave distance-angle heat map, The coordinate system of the target or measuring point under the microwave transceiver antenna Y coordinate, The coordinate system of the target or measuring point under the microwave transceiver antenna X coordinate; Step S2.3: Determine the three-dimensional coordinates of three or more reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as: ， ； in, , is the three-dimensional coordinate of the reference target in the camera coordinate system; is the three-dimensional coordinate of the reference target in the microwave transceiver antenna coordinate system; Obtain the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system and translation matrices .

2. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 1, characterized in that: The step S1 includes fixing the microwave transceiver and the stereo camera, selecting three or more non-collinear targets or measuring points that exist in the field of view of the stereo camera and the microwave transceiver, or placing calibration reference objects as reference targets.

3. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 2, characterized in that: The calibration reference includes greater than or equal to three non-collinear targets that can be simultaneously perceived by the microwave transceiver and the stereo camera. The calibration and matching processing module completes the matching and joint calibration of the targets in the stereo visual imaging and the microwave range-angle heat map, and generates a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

4. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 2, characterized in that: The microwave transceiver includes transmitting and receiving electromagnetic wave signals and outputting baseband signals; the stereo camera includes a binocular camera, a structured light camera and a TOF camera, and the stereo camera uses stereo vision to output relevant information; the relevant information includes information including the parallax or depth of the target.

5. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 1, characterized in that: The step S3 includes identifying and selecting the target or measuring point to be measured in the pixel coordinate system and completing the conversion to the microwave distance-angle thermal map coordinate system; selecting a target or measuring point to be measured in the pixel plane of one of the RGB cameras of the stereo camera ( u tgt , v tgt )in is the pixel value of the target or measuring point in the horizontal direction, is the pixel value of the target or measuring point in the vertical direction; and uniquely determines the target or measuring point in the camera coordinate system Z The coordinate value is , and then determine its three-dimensional coordinates in the camera coordinate system ( x c_tgt ,y c_tgt ,z c_tgt ); its three-dimensional coordinates in the microwave transceiver antenna coordinate system : ； in, is the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system, is the translation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system; The three-dimensional coordinates of the target or measuring point in the microwave transceiver antenna coordinate system Converted into microwave distance-angle heat map: ； in, The distance value of the target to be measured or the measuring point in the microwave distance-angle thermal map; It is the angle value of the target or measuring point in the microwave distance-angle heat map.

6. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 1, characterized in that: The step S3 also includes the conversion of microwave distance-angle heat map coordinates to pixel coordinates, based on the rotation matrix of the measurement point from the microwave transceiver antenna coordinate system to the camera coordinate system and translation matrices , let the coordinates of the target or measuring point in the microwave distance-angle thermal map be , determine the target or measurement point to be measured in the microwave transceiver antenna coordinate system Z Coordinate value , then the target or measuring point to be measured corresponds to X 、 Y The coordinates are: ； in, is the distance value of the target or measuring point in the microwave distance-angle thermal map, is the angle value of the target or measuring point in the microwave distance-angle heat map, The coordinate system of the target or measuring point under the microwave transceiver antenna Y coordinate; The coordinate system of the target or measuring point under the microwave transceiver antenna X coordinate; Assume that the coordinates of the target or measuring point to be measured in the camera coordinate system are , then: ； Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are , the coordinates of the target or measuring point in the pixel coordinate system are ,have: ； in, is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt is the pixel value of the target or measuring point in the vertical direction, is the focal length of the RGB camera.

7. A system for matching and selecting measurement points using stereo vision and microwave thermal imaging, characterized in that: include: Module M1: collects information within the field of view and determines the reference target; Module M2: Construct a coordinate system and establish a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system; Module M3: Matching and selecting targets or measuring points; The module M2 Includes the following submodules: Module M2.1: Complete the reference target in the camera coordinate system Positioning under the camera; establish the camera coordinate system with the optical center of a RGB camera of the stereo camera as the origin ; in, The plane is parallel to the imaging plane of the RGB camera, The positive direction is from the optical center of the RGB camera to the outside; Based on the depth measurement principle of stereo vision, the depth information of the reference target is obtained as the Z coordinate; the stereo vision includes the binocular parallax principle and the light pulse flight time method 3D imaging; based on the pinhole imaging principle, the reference target is calculated X Coordinates and Y coordinate; Module M2.2: Complete the reference target in the microwave transceiver antenna coordinate system Positioning under; Assume that the antenna plane of the microwave transceiver is flat; Among them, the arrangement direction of the antenna array is Axis direction, when looking at the antenna plane from the front, The positive axis points to the right, The positive axis points upwards, The positive axis points out of the antenna plane, the microwave transceiver antenna coordinate system The origin is the position of the first channel; For a target in the microwave imaging range-angle heat map , determine the target or measurement point to be measured in the microwave transceiver antenna coordinate system Z The coordinates are , then the target or measuring point corresponds to X 、 Y The coordinates are: ； in, is the distance value of the target or measuring point in the microwave distance-angle thermal map, is the angle value of the target or measuring point in the microwave distance-angle heat map, The coordinate system of the target or measuring point under the microwave transceiver antenna Y coordinate, The coordinate system of the target or measuring point under the microwave transceiver antenna X coordinate; Module M2.3: Determine the three-dimensional coordinates of three or more reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as: ， ； in, , is the three-dimensional coordinate of the reference target in the camera coordinate system; is the three-dimensional coordinate of the reference target in the microwave transceiver antenna coordinate system; Obtain the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system and translation matrices .

8. The system for matching and selecting measurement points of stereo vision and microwave thermal imaging according to claim 7, characterized in that: The module M1 includes a fixed microwave transceiver and a stereo camera, and selects more than or equal to three non-collinear targets or measurement points that exist in the field of view of the stereo camera and the microwave transceiver, or places calibration reference objects as reference targets.

9. The system for matching and selecting measurement points of stereo vision and microwave thermal imaging according to claim 8, characterized in that: The calibration reference includes greater than or equal to three non-collinear targets that can be simultaneously perceived by the microwave transceiver and the stereo camera. The calibration and matching processing module completes the matching and joint calibration of the targets in the stereo visual imaging and the microwave range-angle heat map, and generates a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

Images