Stereoscopic vision and microwave heat map imaging measuring point matching selection method and system

Abstract

The invention provides a stereo vision and microwave heat map imaging measuring point matching selection method and system, and the method comprises the steps: S1, collecting information in a view field, and determining a reference target; s2, constructing a coordinate system, and establishing a mapping relation from a camera coordinate system to a microwave transceiver antenna coordinate system; and S3, matching and selecting a target or a measuring point. According to the method, the problems that in the prior art, microwave distance-angle heat map measuring points are not intuitively selected, the degree of dependence on priori knowledge is high, and accurate matching with structural feature points is difficult are solved, and three-dimensional space positioning is carried out on targets or measuring points in a view field through stereoscopic vision; the functions of selecting the measuring points from the visual pixel coordinates and accurately mapping the measuring points to the corresponding microwave distance-angle heat map measuring points are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of microwave thermal imaging measurement point identification, matching and selection, and in particular to a method and system for matching and selecting measurement points of stereoscopic vision and microwave thermal imaging. Background Art

[0002] Microwave sensing technology plays an important role in the fields of military, communication, radar, etc. with its advantages of strong penetration, high precision, anti-interference ability and strong environmental adaptability. It can work stably in complex environments, providing effective solutions for fine measurement and covert perception, and has become an indispensable key technology in the field of modern science and technology.

[0003] Microwave transceivers mainly rely on the reflection of electromagnetic waves by the target when sensing the target. The premise of micro-motion sensing based on microwaves is to accurately locate and select the measuring points. Using a multi-transmitter and multi-receiver linear frequency modulation continuous wave microwave transceiver, multiple targets of different distance dimensions can be distinguished by the beat frequency, and the angle can be further distinguished by the time difference from the same target to different receiving antennas, so as to obtain the distance-angle joint positioning of the full-field target or measuring point. The identification and recognition of the measuring point rely on the distance-angle thermal image imaging of the microwave transceiver. The selection of measuring points from the thermal image is heavily dependent on the tester's prior knowledge of the specific spatial distribution information of the measured target or measuring point. When the test scene is more complex, the measuring points are irregularly arranged and the number is large, or the test distance is far, the efficiency and accuracy of the measuring point selection are low, or even fail, which restricts the wide application of technologies including full-field vibration and deformation displacement measurement based on microwave sensing in engineering practice. Therefore, there is an urgent need for an intuitive and easy-to-operate microwave imaging thermal image measuring point identification, positioning and selection technology and method.

[0004] In the Chinese patent document with the publication number CN217029797U, a damping type microwave transceiver is disclosed, which includes a mounting seat, a damping assembly, a microwave transceiver body, a limiting column, a heat sink and a wiring port. The bottom of the damping assembly is fixedly connected to the mounting seat, the top of the damping assembly is fixedly connected to the microwave transceiver body, the top of the mounting seat is extended upwardly and provided with limiting blocks located on the left and right sides of the microwave transceiver body, the limiting blocks are provided with limiting openings, the limiting columns are passed through the limiting openings and are threadedly connected to the microwave transceiver body, the heat sink is installed on the top of the microwave transceiver body, the front end surface of the microwave transceiver body is provided with a wiring port, the interior of the microwave transceiver body is provided with a control host, and the wiring port is electrically connected to the control host. The patent document provides a microwave transceiver with good damping performance and easy to install and fix, while the present invention is to solve the problem that the distance-angle heat map of the microwave transceiver is not intuitive and difficult to match with the structural feature points, which is essentially different from the technical problem to be solved by the patent document. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for matching and selecting measurement points of stereo vision and microwave thermal imaging.

[0006] A method for selecting and matching stereo vision and microwave thermal imaging measurement points provided by the present invention includes:

[0007] Step S1: Collect information within the field of view and determine the reference target;

[0008] Step S2: constructing a coordinate system and establishing a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system;

[0009] Step S3: Match and select targets or measuring points.

[0010] Preferably, step S1 includes fixing the microwave transceiver and the stereo camera, selecting greater than or equal to three non-collinear targets or measuring points that exist in the field of view of the stereo camera and the microwave transceiver, or placing a calibration reference object as a reference target.

[0011] Preferably, the calibration reference object includes greater than or equal to three non-collinear targets that can be simultaneously sensed by the microwave transceiver and the stereo camera, and the calibration and matching processing module completes the matching and joint calibration of the targets in the stereoscopic visual imaging and the microwave range-angle heat map to generate a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

[0012] Preferably, the microwave transceiver includes transmitting and receiving electromagnetic wave signals and outputting baseband signals; the stereo camera includes a binocular camera, a structured light camera and a TOF camera, and the stereo camera uses stereo vision to output relevant information; the relevant information includes information including parallax or depth of the target.

[0013] Preferably, step S2 includes the following sub-steps:

[0014] Step S2.1: Complete the reference target in the camera coordinate system OX c Y c Z c Positioning under; Establish the camera coordinate system OX with the optical center of a RGB camera of the stereo camera as the origin c Y c Z c ;

[0015] Among them, X c OY c The plane is parallel to the imaging plane of the RGB camera, Z c The positive direction is from the optical center of the RGB camera to the outside;

[0016] Based on the depth measurement principle of stereoscopic vision, the depth information of the reference target is obtained as the Z coordinate; the stereoscopic vision includes the binocular parallax principle and the light pulse flight time method 3D imaging; based on other principles, the X coordinate and Y coordinate of the reference target are calculated; the other principles include the pinhole imaging principle;

[0017] Step S2.2: Complete the reference target in the microwave transceiver antenna coordinate system OX r Y r Z r Positioning under; Assume the antenna plane of the microwave transceiver is Y r OZ r flat;

[0018] Among them, the arrangement direction of the antenna array is Y r Axis direction, when looking at the antenna plane from the front, Y r The positive axis points to the right, Z r The positive axis points upward, X r The positive axis points out of the antenna plane, and the microwave transceiver antenna coordinate system OX r Y r Z r The origin is the position of the first channel;

[0019] For a target (R, θ) in the microwave imaging range-angle heat map, determine the Z coordinate of the measuring point in the microwave transceiver antenna coordinate system as z r , then the X and Y coordinates corresponding to the measuring point are:

[0020]

[0021] Where R is the distance value of the target in the microwave distance-angle heat map, θ is the angle value of the target in the microwave distance-angle heat map, and y r is the Y coordinate of the target in the microwave transceiver antenna coordinate system, x r is the X coordinate of the target in the microwave transceiver antenna coordinate system;

[0022] Step S2.3: Determine the three-dimensional coordinates of greater than or equal to three reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as:

[0023]

[0024] Where n≥3, P cam is the three-dimensional coordinate of the reference target in the camera coordinate system; P radar is the three-dimensional coordinate of the reference target in the microwave transceiver antenna coordinate system;

[0025] Obtain the rotation matrix R and translation matrix T from the camera coordinate system to the microwave transceiver antenna coordinate system.

[0026] Preferably, the step S3 includes identifying and selecting the target or measuring point to be measured in the pixel coordinate system, and completing the conversion to the microwave distance-angle heat map coordinates; selecting a target or measuring point to be measured (u tgt ,v tgt ), where u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt is the pixel value of the target or measuring point in the vertical direction; and uniquely determines the Z coordinate value of the target or measuring point in the camera coordinate system as z c_tgt , and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt ); its three-dimensional coordinates in the microwave transceiver antenna coordinate system (x r_tgt ,y r_tgt ,z r_tgt ):

[0027]

[0028] Where R is the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system, and T is the translation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system;

[0029] The three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ) into a microwave distance-angle heat map:

[0030]

[0031] Among them, R tgt is the distance value of the target or measuring point in the microwave distance-angle thermal map; θ tgt It is the angle value of the target or measuring point in the microwave distance-angle heat map.

[0032] Preferably, step S3 further includes conversion of microwave distance-angle heat map coordinates to pixel coordinates, based on the rotation matrix R' and translation matrix T' of the measuring point from the microwave transceiver antenna coordinate system to the camera coordinate system, assuming that the coordinates of the target to be measured in the microwave distance-angle heat map are (R tgt ,θ tgt ), determine the Z coordinate value z of the target or measuring point in the microwave transceiver antenna coordinate system r_tgt , then the X and Y coordinates of the target or measurement point in the microwave transceiver antenna coordinate system are:

[0033]

[0034] Among them, R tgt is the distance value of the target in the microwave distance-angle heat map, θ tgt is the angle value of the target in the microwave distance-angle heat map, y r_tgt is the Y coordinate of the target or measuring point in the microwave transceiver antenna coordinate system; r_tgt is the X coordinate of the target or measuring point in the microwave transceiver antenna coordinate system;

[0035] Assume that the coordinates of the target or measuring point in the camera coordinate system are (x c_tgt ,y c_tgt ,z c_tgt ), then:

[0036]

[0037] Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are (u cpoint ,v cpoint ), the coordinates of the target or measuring point in the pixel coordinate system are (u tgt ,v tgt ),have:

[0038]

[0039] Among them, u cpoint is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, v cpoint is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point to be converted in the horizontal direction, v tgt is the pixel value of the target or measurement point to be converted in the vertical direction, and f is the focal length of the RGB camera.

[0040] A system for matching and selecting measurement points of stereo vision and microwave thermal imaging provided by the present invention comprises:

[0041] Module M1: collect information within the field of view and determine the reference target;

[0042] Module M2: Construct a coordinate system and establish a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system;

[0043] Module M3: Matching and selecting targets or measuring points.

[0044] Preferably, the module M1 includes a fixed microwave transceiver and a stereo camera, and selects more than or equal to three non-collinear targets or measuring points that exist in the field of view of the stereo camera and the microwave transceiver, or places calibration reference objects as reference targets.

[0045] Preferably, the calibration reference object includes greater than or equal to three non-collinear targets that can be simultaneously sensed by the microwave transceiver and the stereo camera, and the calibration and matching processing module completes the matching and joint calibration of the targets in the stereoscopic visual imaging and the microwave range-angle heat map to generate a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

[0046] Preferably, the microwave transceiver includes transmitting and receiving electromagnetic wave signals and outputting baseband signals; the stereo camera includes a binocular camera, a structured light camera and a TOF camera, and the stereo camera uses stereo vision to output relevant information; the relevant information includes information including parallax or depth of the target.

[0047] Preferably, the module M2 includes the following submodules:

[0048] Module M2.1: Complete the reference target in the camera coordinate system OX c Y c Z c Positioning under; Establish the camera coordinate system OX with the optical center of a RGB camera of the stereo camera as the origin c Y c Z c ;

[0049] Among them, X c OY c The plane is parallel to the imaging plane of the RGB camera, Z c The positive direction is from the optical center of the RGB camera to the outside;

[0050] Based on the depth measurement principle of stereoscopic vision, the depth information of the reference target is obtained as the Z coordinate; the stereoscopic vision includes the binocular parallax principle and the light pulse flight time method 3D imaging; based on other principles, the X coordinate and Y coordinate of the reference target are calculated; the other principles include the pinhole imaging principle;

[0051] Module M2.2: Complete the reference target in the microwave transceiver antenna coordinate system OX r Y r Z r Positioning under; Assume the antenna plane of the microwave transceiver is Y r OZ r flat;

[0052] Among them, the arrangement direction of the antenna array is Y r Axis direction, when looking at the antenna plane from the front, Yr The positive axis points to the right, Z r The positive axis points upward, X r The positive axis points out of the antenna plane, and the microwave transceiver antenna coordinate system OX r Y r Z r The origin is the position of the first channel;

[0053] For a target (R, θ) in the microwave imaging range-angle heat map, determine the Z coordinate of the measuring point in the microwave transceiver antenna coordinate system as z r , then the X and Y coordinates corresponding to the measuring point are:

[0054]

[0055] Where R is the distance value of the target in the microwave distance-angle heat map, θ is the angle value of the target in the microwave distance-angle heat map, and y r is the Y coordinate of the target in the microwave transceiver antenna coordinate system, x r is the X coordinate of the target in the microwave transceiver antenna coordinate system;

[0056] Module M2.3: Determine the three-dimensional coordinates of more than or equal to three reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as:

[0057]

[0058] Where n≥3, P cam is the three-dimensional coordinate of the reference target in the camera coordinate system; P radar is the three-dimensional coordinate of the reference target in the microwave transceiver antenna coordinate system;

[0059] Obtain the rotation matrix R and translation matrix T from the camera coordinate system to the microwave transceiver antenna coordinate system.

[0060] Preferably, the module M3 includes identifying and selecting the target or measuring point to be measured in the pixel coordinate system, and completing the conversion to the microwave distance-angle heat map coordinate system; selecting a target or measuring point to be measured (u tgt ,v tgt ), where u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt is the pixel value of the target or measuring point in the vertical direction; and uniquely determines the Z coordinate value of the target or measuring point in the camera coordinate system as z c_tgt , and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt); its three-dimensional coordinates in the microwave transceiver antenna coordinate system (x r_tgt ,y r_tgt ,z r_tgt ):

[0061]

[0062] Where R is the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system, and T is the translation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system;

[0063] The three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ) into a microwave distance-angle heat map:

[0064]

[0065] Among them, R tgt is the distance value of the target or measuring point in the microwave distance-angle thermal map; θ tgt It is the angle value of the target or measuring point in the microwave distance-angle heat map.

[0066] Preferably, the module M3 further includes a conversion of microwave distance-angle heat map coordinates to pixel coordinates, based on the rotation matrix R' and translation matrix T' of the measuring point from the microwave transceiver antenna coordinate system to the camera coordinate system, assuming that the coordinates of the target to be measured in the microwave distance-angle heat map are (R tgt ,θ tgt ), determine the Z coordinate value z of the target or measuring point in the microwave transceiver antenna coordinate system r_tgt , then the X and Y coordinates of the target or measurement point in the microwave transceiver antenna coordinate system are:

[0067]

[0068] Among them, R tgt is the distance value of the target in the microwave distance-angle heat map, θ tgt is the angle value of the target in the microwave distance-angle heat map, y r_tgt is the Y coordinate of the target or measuring point in the microwave transceiver antenna coordinate system; r_tgt is the X coordinate of the target or measuring point in the microwave transceiver antenna coordinate system;

[0069] Assume that the coordinates of the target or measuring point in the camera coordinate system are (x c_tgt ,y c_tgt ,z c_tgt ), then:

[0070]

[0071] Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are (u cpoint ,v cpoint ), the coordinates of the target or measuring point in the pixel coordinate system are (u tgt ,v tgt ),have:

[0072]

[0073] Among them, u cpoint is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, v cpoint is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point to be converted in the horizontal direction, v tgt is the pixel value of the target or measurement point to be converted in the vertical direction, and f is the focal length of the RGB camera.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. The present invention overcomes the difficulties in the prior art of microwave distance-angle heat map measurement point selection, which is not intuitive, highly dependent on prior knowledge, and difficult to accurately match with structural feature points. It uses stereo vision to perform three-dimensional spatial positioning of targets or measurement points in the field of view, and realizes the function of selecting measurement points from visual pixel coordinates and accurately mapping them to corresponding microwave distance-angle heat map measurement points.

[0076] 2. The present invention improves the convenience and efficiency of microwave distance-angle thermal map measurement point selection, realizes the measurement point matching and selection of the fusion of stereo vision and microwave thermal map imaging, and provides a technical means for the microwave distance-angle thermal map measurement point coordinate selection and structural coordinate matching in complex scenes.

[0077] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0079] Figure 1 The figure is a flow chart of the method of the present invention.

[0080] Figure 2It is a schematic diagram of the matching of stereoscopic vision imaging and microwave transceiver heat map measurement points in an embodiment of the present invention.

[0081] Figure 3 Schematic diagram of the process of establishing a camera coordinate system in an embodiment of the present invention.

[0082] Figure 4 Schematic diagram of the process of establishing a microwave transceiver antenna coordinate system in an embodiment of the present invention.

[0083] Figure 5 It is a system structure block diagram of the present invention. DETAILED DESCRIPTION

[0084] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0085] The existing microwave thermal map measurement point identification and selection rely on the tester's prior knowledge of the specific spatial distribution information of the target structure or measurement point and the microwave perception principle. When the test scene is complex, the measurement points are irregularly arranged and the number is large, the efficiency and accuracy of the measurement point identification and selection will be greatly reduced. The more abstract thermal map measurement points are often difficult to match with the structural feature points. In order to solve this problem, it is necessary to measure the target; and in order to collect measurement data, the target measurement point needs to be located first. How to locate the target measurement point? By using a microwave transceiver and a stereo camera and constructing a camera coordinate system and a microwave transceiver antenna coordinate system respectively, the three-dimensional coordinates of the target measurement point are obtained. At this time, although different points to be measured can be expressed intuitively, the conversion of the coordinates of the target measurement point in different coordinate systems cannot be completed. Therefore, it is necessary to calculate the coordinate transformation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system based on these coordinate points to complete the calibration process; but at this time, it is still impossible to measure the target measuring point; therefore, a pixel coordinate system is first established, the target or measuring point to be measured is selected in the RGB camera pixel plane, the three-dimensional coordinates in the camera coordinate system are determined, and the coordinate transformation matrix is ​​used to convert it to the microwave transceiver antenna coordinate system, and then converted to the microwave distance-angle heat map. Through calibration and measurement, the identification and selection of the target or measuring point to be measured in the pixel plane and the matching conversion of pixel coordinates to the microwave distance-angle heat map are completed; for the convenience of calculation, the present invention can also realize the conversion of microwave distance-angle heat map coordinates to pixel coordinates.

[0086] Reference Figure 1 and Figure 2As shown, first, fix the microwave transceiver and the stereo camera, select more than or equal to three non-collinear targets that exist in the field of view of the stereo camera and the microwave transceiver, or place calibration reference objects as reference targets, and determine the corresponding three-dimensional coordinate point pairs of these reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, and then calculate the coordinate transformation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system based on these coordinate point pairs to complete the calibration process; then, select the target or measuring point to be measured in the RGB camera pixel plane, determine its three-dimensional coordinates in the camera coordinate system, and use the coordinate transformation matrix obtained in the calibration process to transform its coordinates into the microwave transceiver antenna coordinate system, and then further transform it into the microwave distance-angle heat map to complete the matching and selection of the target or measuring point.

[0087] The above is a basic embodiment of the present invention. The technical solution of the present invention is further described below through a preferred embodiment.

[0088] Example 1

[0089] Reference Figure 1 As shown, a method for selecting and matching measurement points of stereo vision and microwave thermal imaging includes:

[0090] Step 1: Calibration of the coordinate transformation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system:

[0091] First, the calibration scene layout is performed:

[0092] Fix the microwave transceiver and stereo camera to keep them relatively still and make their fields of view cover the area where the target or measurement point is located. Select three or more non-collinear targets in the field of view that exist in both the microwave transceiver and stereo camera imaging or use calibration reference objects as reference targets.

[0093] Secondly, the reference target is in the camera coordinate system OX c Y c Z c Positioning below.

[0094] Reference Figure 3 As shown, the camera coordinate system OX is established with the optical center of one of the RGB cameras of the stereo camera as the origin. c Y c Z c , where X c OY c The plane is parallel to the imaging plane of the RGB camera, Z cThe positive direction points from the optical center of the RGB camera to the outside. With the help of stereo vision including binocular parallax principle, light pulse flight time method and other depth measurement principles, the depth information of the reference target is obtained, that is, the Z coordinate of the reference target in the above camera coordinate system, and based on methods including pinhole imaging principle, the X and Y coordinates of the reference target in the above coordinate system are calculated.

[0095] In this embodiment, two cameras with parallel optical axes, consistent focal lengths and located in the same plane are used to form a binocular camera. For a certain target or measuring point, the imaging coordinates on the left camera pixel plane and the right camera pixel plane are respectively (u l ,v l ) and (u r ,v r ), the coordinates of the left camera and the right camera in the image plane coordinate system are (x l ,y l ) and (x r ,y r ), establish the camera coordinate system OX with the optical center of the left camera as the origin c Y c Z c , then the coordinates of the target or measuring point in the camera coordinate system (x c ,y c ,z c )for:

[0096]

[0097] Wherein, b is the baseline distance of the above binocular cameras, and f is the focal length of the two cameras.

[0098] Then the reference target is located in the microwave transceiver antenna coordinate system OX r Y r Z r Positioning below.

[0099] Reference Figure 4 As shown, assume that the antenna plane of the microwave transceiver is Y r OZ r Plane, where the arrangement direction of the linear antenna array is Y r When looking at the antenna plane from the front, Y r The positive axis points to the right, Z r The positive axis points upward, X r The positive axis points out of the antenna plane, and the antenna coordinate system OX r Y r Z r The origin is the position of the first virtual channel, that is, the midpoint of the line connecting the first transmitting antenna and the first receiving antenna.

[0100] For a target (R, θ) in the microwave imaging range-angle heat map, the Z coordinate of the measuring point in the microwave transceiver antenna coordinate system is determined by measurement or prior knowledge as z r , then the X and Y coordinates corresponding to the measuring point are:

[0101]

[0102] Where R is the distance value of the target in the microwave distance-angle heat map, θ is the angle value of the target in the microwave distance-angle heat map, and y r is the Y coordinate of the target in the microwave transceiver antenna coordinate system, x r is the X coordinate of the target in the microwave transceiver antenna coordinate system. r Negative values ​​are discarded because the front of the microwave transceiver antenna plane is X r Axis positive direction, the rear of the microwave transceiver antenna plane is X r In the negative direction of the axis, the target can be sensed by the microwave transceiver only if it is in front of the antenna plane.

[0103] The mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system is established by means of coordinate transformation matrix:

[0104] Determine the three-dimensional coordinates of more than or equal to three reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as:

[0105]

[0106] Where n≥3, P cam are the three-dimensional coordinates of these targets in the camera coordinate system; P radar are the three-dimensional coordinates of these targets in the microwave transceiver antenna coordinate system.

[0107] The rotation matrix R and translation matrix T from the camera coordinate system to the microwave transceiver antenna coordinate system are obtained by the singular value decomposition method. The specific process is:

[0108] Assume that the average coordinates of these targets in the camera coordinate system are The average coordinate in the microwave transceiver antenna coordinate system is Right now:

[0109]

[0110] Constructing the Matrix in Indicates P cam The i-th column of Indicates P radar The i-th column of . Perform singular value decomposition of the matrix M:

[0111] M=UΣV T ;

[0112] Then the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system is:

[0113] R=VU T ;

[0114] The translation matrix is:

[0115]

[0116] Step 2: Identify and select the target or measurement point in the pixel coordinate system, and complete the conversion to the microwave distance-angle thermal map coordinates.

[0117] Select a target or point to be measured on the pixel plane of one of the RGB cameras of the stereo camera (u tgt ,v tgt ), where u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt is the pixel value of the target or measuring point in the vertical direction; by referring to the target in the camera coordinate system OX c Y c Z c The positioning under the camera coordinate system uniquely determines its depth information, that is, the Z coordinate value z of the target or measuring point in the camera coordinate system c_tgt , and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt Then, the obtained rotation matrix R and translation matrix T are used to obtain its three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ).

[0118] For a target or measurement point (u l_tgt ,v l_tgt ), its coordinates (x l_tgt ,y l_tgt ) and (x r_tgt ,y r_tgt ), and then determine its three-dimensional coordinates (x c_tgt ,y c_tgt ,z c_tgt Then, the obtained rotation matrix R and translation matrix T are used to obtain its three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ):

[0119]

[0120] Finally, the three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ) into a microwave distance-angle heat map:

[0121]

[0122] Among them, R tgt is the distance value of the target in the microwave distance-angle heat map, θ tgt is the angle value of the target in the microwave distance-angle heat map, thus completing the pixel plane coordinates (u l_tgt ,v l_tgt ) to the microwave distance-angle heat map coordinates (R tgt ,θ tgt ) mapping and conversion.

[0123] In addition, the present invention can also realize the conversion of microwave distance-angle heat map coordinates to pixel coordinates, and the specific method is as follows:

[0124] Still using the three-dimensional coordinates of the reference target obtained above in the microwave transceiver antenna coordinate system and the camera coordinate system, construct the matrix Calculate the rotation matrix R' and translation matrix T' of the measurement point from the microwave transceiver antenna coordinate system to the camera coordinate system.

[0125] Assume that the coordinates of the target to be measured in the microwave distance-angle heat map are (R tgt ,θ tgt ), through measurement or prior knowledge, determine the Z coordinate of the target or measurement point to be measured in the microwave transceiver antenna coordinate system as z r_tgt , then the X and Y coordinates of the target or measurement point in the microwave transceiver antenna coordinate system are:

[0126]

[0127] Among them, y r_tgt is the Y coordinate of the target or measuring point in the microwave transceiver antenna coordinate system; r_tgt is the X coordinate of the target or measurement point in the microwave transceiver antenna coordinate system.

[0128] Assume that the coordinates of the target or measuring point in the camera coordinate system are (x c_tgt ,y c_tgt ,z c_tgt ), then:

[0129]

[0130] Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are (u cpoint ,v cpoint ), the coordinates of the target or measuring point in the pixel coordinate system are (u tgt ,v tgt ),have:

[0131]

[0132] Among them, u cpoint is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, v cpoint is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point to be converted in the horizontal direction, v tgt is the pixel value of the target or measurement point to be converted in the vertical direction, and f is the focal length of the RGB camera.

[0133] This completes the measurement of the target or measuring point in the microwave distance-angle thermal map coordinates (R tgt ,θ tgt ) to the RGB camera pixel plane coordinates (u tgt ,v tgt ) mapping and conversion.

[0134] The present invention overcomes the difficulties of the prior art microwave distance-angle heat map measurement point selection that is not intuitive, highly dependent on prior knowledge, and difficult to accurately match with structural feature points. It uses stereoscopic vision to perform three-dimensional spatial positioning of targets or measurement points in the field of view, and realizes the function of selecting measurement points from visual pixel coordinates and accurately mapping them to corresponding microwave distance-angle heat map measurement points, thereby improving the convenience and efficiency of microwave distance-angle heat map measurement point selection, realizing measurement point matching and selection that integrates stereoscopic vision and microwave heat map imaging, and providing technical means for microwave distance-angle heat map measurement point coordinate selection and structural coordinate matching in complex scenes.

[0135] Reference Figure 5 As shown, the present invention also provides a system for matching and selecting measurement points of stereo vision and microwave thermal imaging, comprising:

[0136] Microwave transceiver, stereo camera, calibration reference, signal acquisition module, calibration and matching processing module and display and storage module.

[0137] Among them, the microwave transceiver is used to transmit and receive electromagnetic wave signals, output baseband signals, and realize distance-angle thermal map positioning and imaging.

[0138] Stereo camera: Uses stereo vision modalities including binocular cameras, structured light cameras, and TOF cameras to output information including parallax or depth of the target.

[0139] Calibration reference objects: include greater than or equal to three non-collinear targets that can be simultaneously sensed by the microwave transceiver and the stereo camera, and are used for joint calibration and matching of targets in stereo vision imaging and microwave range-angle heat maps.

[0140] Signal acquisition module: used to collect baseband signals from microwave transceivers, as well as raw images and disparity or depth information from stereo cameras.

[0141] The calibration and matching processing module includes the following processing contents: processing the baseband signal output by the microwave transceiver, obtaining the distance-angle heat map, and converting the coordinates of the target or the measuring point to the microwave transceiver antenna coordinate system; locating the three-dimensional coordinates of the target in the camera coordinate system according to the parallax or depth information output by the stereo vision module; calibrating and generating the coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system; and calculating the corresponding microwave distance-angle heat map coordinate values ​​obtained by conversion according to the coordinate transformation matrix and the pixel point coordinates selected in the RGB camera pixel coordinate system.

[0142] The display and storage module processes the following: display and output of stereo camera visual imaging, depth map and microwave distance-angle thermal map information for users to calibrate and select and match measurement points; display the coordinates of the selected pixel points and simultaneously display the corresponding microwave distance-angle thermal map coordinates; store relevant coordinate information and visual image, depth map, microwave distance-angle thermal map imaging information.

[0143] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0144] In the description of the present application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0145] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for selecting and matching measurement points of stereo vision and microwave thermal imaging, characterized in that: include: Step S1: Collect information within the field of view and determine the reference target; Step S2: constructing a coordinate system and establishing a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system; Step S3: Match and select targets or measuring points.

2. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 1, characterized in that: The step S1 includes fixing the microwave transceiver and the stereo camera, selecting more than or equal to three non-collinear targets or measuring points that exist in the field of view of the stereo camera and the microwave transceiver, or placing a calibration reference object as a reference target.

3. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 2, characterized in that: The calibration reference includes more than or equal to three non-collinear targets that can be simultaneously sensed by the microwave transceiver and the stereo camera. The calibration and matching processing module completes the matching and joint calibration of the targets in the stereoscopic visual imaging and the microwave distance-angle heat map, and generates a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

4. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 2, characterized in that: The microwave transceiver includes transmitting and receiving electromagnetic wave signals and outputting baseband signals; the stereo camera includes a binocular camera, a structured light camera and a TOF camera, and the stereo camera uses stereo vision to output relevant information; the relevant information includes information including the parallax or depth of the target.

5. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 2, characterized in that: The step S2 comprises the following sub-steps: Step S2.1: Complete the reference target in the camera coordinate system OX c Y c Z c Positioning under; Establish the camera coordinate system OX with the optical center of a RGB camera of the stereo camera as the origin c Y c Z c ; Among them, X c OY c The plane is parallel to the imaging plane of the RGB camera, Z c The positive direction is from the optical center of the RGB camera to the outside; Based on the depth measurement principle of stereoscopic vision, the depth information of the reference target is obtained as the Z coordinate; the stereoscopic vision includes the binocular parallax principle and the light pulse flight time method 3D imaging; based on other principles, the X coordinate and Y coordinate of the reference target are calculated; the other principles include the pinhole imaging principle; Step S2.2: Complete the reference target in the microwave transceiver antenna coordinate system OX r Y r Z r Positioning under; Assume the antenna plane of the microwave transceiver is Y r OZ r flat; Among them, the arrangement direction of the antenna array is Y r Axis direction, when looking at the antenna plane from the front, Y r The positive axis points to the right, Z r The positive axis points upward, X r The positive axis points out of the antenna plane, and the microwave transceiver antenna coordinate system OX r Y r Z r The origin is the position of the first channel; For a target (R, θ) in the microwave imaging range-angle heat map, determine the Z coordinate of the measuring point in the microwave transceiver antenna coordinate system as z r , then the X and Y coordinates corresponding to the measuring point are: Where R is the distance value of the target in the microwave distance-angle heat map, θ is the angle value of the target in the microwave distance-angle heat map, and y r is the Y coordinate of the target in the microwave transceiver antenna coordinate system, x r is the X coordinate of the target in the microwave transceiver antenna coordinate system; Step S2.3: Determine the three-dimensional coordinates of greater than or equal to three reference targets in the camera coordinate system and the microwave transceiver antenna coordinate system, which are recorded as: Where n≥3, P cam is the three-dimensional coordinate of the reference target in the camera coordinate system; P radar is the three-dimensional coordinate of the reference target in the microwave transceiver antenna coordinate system; Obtain the rotation matrix R and translation matrix T from the camera coordinate system to the microwave transceiver antenna coordinate system.

6. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 5, characterized in that: The step S3 includes identifying and selecting the target or measuring point to be measured in the pixel coordinate system, and completing the conversion to the microwave distance-angle heat map coordinates; selecting a target or measuring point (utgt, vtgt) to be measured in the pixel plane of one of the RGB cameras of the stereo camera, where u tgt is the pixel value of the target or measuring point in the horizontal direction, v tgt is the pixel value of the target or measuring point in the vertical direction; and uniquely determines the Z coordinate value of the target or measuring point in the camera coordinate system as z c_tgt , and then determine its three-dimensional coordinates (xc_tgt, yc_tgt, zc_tgt) in the camera coordinate system; its three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ): Where R is the rotation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system, and T is the translation matrix from the camera coordinate system to the microwave transceiver antenna coordinate system; The three-dimensional coordinates (x r_tgt ,y r_tgt ,z r_tgt ) into a microwave distance-angle heat map: Among them, R tgt is the distance value of the target or measuring point in the microwave distance-angle thermal map; θ tgt It is the angle value of the target or measuring point in the microwave distance-angle heat map.

7. The method for selecting and matching stereo vision and microwave thermal imaging measurement points according to claim 5, characterized in that: The step S3 also includes the conversion of microwave distance-angle heat map coordinates to pixel coordinates, based on the rotation matrix R' and translation matrix T' of the measuring point from the microwave transceiver antenna coordinate system to the camera coordinate system, assuming that the coordinates of the target to be measured in the microwave distance-angle heat map are (R tgt ,θ tgt ), determine the Z coordinate value z of the target or measuring point in the microwave transceiver antenna coordinate system r_tgt , then the target or measuring point is in the microwave transceiver The corresponding X and Y coordinates in the antenna coordinate system are: Among them, R tgt is the distance value of the target in the microwave distance-angle heat map, θ tgt is the angle value of the target in the microwave distance-angle heat map, y r_tgt is the Y coordinate of the target or measuring point in the microwave transceiver antenna coordinate system; r_tgt is the X coordinate of the target or measuring point in the microwave transceiver antenna coordinate system; Assume that the coordinates of the target or measuring point in the camera coordinate system are (x c_tgt ,y c_tgt ,z c_tgt ), then: Assume that the pixel coordinates corresponding to the origin of the image coordinate system of the RGB camera to be converted are (u cpoint ,v cpoint ), the coordinates of the target or measuring point in the pixel coordinate system are (u tgt ,v tgt ),have: Among them, u cpoint is the pixel value in the horizontal direction of the origin of the image coordinate system of the RGB camera to be converted, v cpoint is the pixel value in the vertical direction of the origin of the image coordinate system of the RGB camera to be converted, u tgt is the pixel value of the target or measuring point to be converted in the horizontal direction, v tgt is the pixel value of the target or measurement point to be converted in the vertical direction, and f is the focal length of the RGB camera.

8. A system for matching and selecting measurement points of stereo vision and microwave thermal imaging, characterized in that: include: Module M1: collect information within the field of view and determine the reference target; Module M2: Construct a coordinate system and establish a mapping relationship between the camera coordinate system and the microwave transceiver antenna coordinate system; Module M3: Matching and selecting targets or measuring points.

9. The system for matching and selecting measurement points of stereo vision and microwave thermal imaging according to claim 8, characterized in that: The module M1 includes a fixed microwave transceiver and a stereo camera, and selects more than or equal to three non-collinear targets or measuring points that exist in the field of view of the stereo camera and the microwave transceiver, or places calibration reference objects as reference targets.

10. The system for matching and selecting measurement points of stereo vision and microwave thermal imaging according to claim 9, characterized in that: The calibration reference includes more than or equal to three non-collinear targets that can be simultaneously sensed by the microwave transceiver and the stereo camera. The calibration and matching processing module completes the matching and joint calibration of the targets in the stereoscopic visual imaging and the microwave distance-angle heat map, and generates a coordinate transformation matrix between the camera coordinate system and the microwave transceiver antenna coordinate system.

Images