Remote space registration method and system of radar photoelectric sensor

Abstract

The invention relates to the technical field of multi-mode registration, and discloses a long-distance space registration method and system for a radar photoelectric sensor, and the method comprises the steps: obtaining an initial conversion matrix between a radar coordinate system and a photoelectric sensor coordinate system; preprocessing a target image acquired by a photoelectric sensor in a preset long-distance range to generate photoelectric image feature information; the radar point cloud is mapped to a photoelectric image plane based on the initial conversion matrix, a visual consistency error is calculated based on photoelectric image feature information, and a digital elevation map is utilized to match elevation information of the radar point cloud to calculate an elevation consistency error; and constructing a joint optimization objective function containing dynamic weight distribution of the visual consistency error term and the elevation consistency error term, and solving optimized conversion parameters by minimizing the joint optimization objective function to realize long-distance spatial registration. The method does not need to depend on long-distance target deployment, the implementation cost and the operation difficulty are reduced, and the engineering practicability and the real-time performance are both considered.

Description

Technical Field

[0001] This invention relates to the field of multimodal registration technology, specifically to a long-range spatial registration method and system for radar photoelectric sensors. Background Technology

[0002] Multimodal registration is a key technology in computer vision, robot perception, and multi-sensor fusion. It aims to achieve data consistency and collaborative processing by establishing mathematical mapping relationships between different sensors or coordinate systems. In fields such as autonomous driving, remote sensing, and industrial inspection, the data collected by sensors often exhibits spatial or temporal misalignment due to differences in installation location, viewing angle, or operating principle. Multimodal registration eliminates these discrepancies by calculating parameters such as rotation, translation, and scaling, providing a foundation for multi-source data fusion.

[0003] Radar-electro-optical sensor registration is a specific application of this technology in the joint registration of heterogeneous sensors (such as millimeter-wave radar and visible / infrared cameras). Its core is solving the transformation matrix (extrinsic parameter matrix) between the radar coordinate system and the opto-optical sensor coordinate system. Since the radar outputs point cloud data while the opto-optical sensor acquires images, the data modes and resolutions of the two differ significantly. Registration is necessary to determine their relative pose (rotation matrix and translation vector). Typical methods include joint registration based on common-view targets (such as corner reflectors) or optimizing the extrinsic parameters through feature association (such as edge and contour matching). The accuracy of the extrinsic parameter matrix directly affects the spatiotemporal alignment of the radar and opto-optical data, thus determining the performance of subsequent tasks.

[0004] Various methods differ in cost and performance, and their applicability varies under different conditions. However, in long-range detection scenarios, these methods face significant increases in cost and accuracy degradation as the target distance increases. Therefore, it is necessary to explore new long-range radar-electro-optical registration methods to achieve better economy and timeliness while ensuring registration accuracy. Summary of the Invention

[0005] In view of this, in order to solve the problem of how to achieve accurate registration of radar photoelectric sensors in long-range detection scenarios while also being economical and timely, this invention provides a long-range spatial registration method and system for radar photoelectric sensors. This method significantly reduces the dependence on the amount of training data while ensuring measurement accuracy; improves the accuracy of long-range registration; simplifies the implementation process; and enhances engineering practicality.

[0006] In a first aspect, the present invention provides a long-range spatial registration method for radar photoelectric sensors, comprising:

[0007] Obtain the initial transformation matrix between the radar coordinate system and the photoelectric sensor coordinate system;

[0008] Preprocess the target image acquired by the photoelectric sensor within a preset long-distance range to generate photoelectric image feature information;

[0009] The radar point cloud is mapped to the photoelectric image plane based on the initial transformation matrix, and the visual consistency error is calculated based on the photoelectric image feature information. The elevation consistency error is calculated by matching the elevation information of the radar point cloud with the digital elevation map.

[0010] A joint optimization objective function is constructed, which includes dynamic weight allocation of visual consistency error term and elevation consistency error term. The transformation parameters are solved by minimizing the joint optimization objective function to achieve long-range spatial registration of radar photoelectric sensors.

[0011] The long-range spatial registration method for radar photoelectric sensors provided in this invention effectively overcomes the limitations of long-distance (hundreds to thousands of meters) registration by acquiring an initial transformation matrix, preprocessing to generate photoelectric image feature information, and constructing a joint optimization objective function based on visual and elevation errors to solve for transformation parameters. It eliminates the need for long-distance target deployment, reducing implementation costs and operational difficulty. Simultaneously, by focusing on key areas and fusing elevation information from digital elevation maps, it enhances the stability of features in long-distance scenarios, mitigating the impact of sparse point clouds and blurred image textures. Combined with a partitioned differentiated registration strategy, it significantly improves registration accuracy and robustness. Furthermore, it eliminates the need for massive amounts of labeled data, balancing engineering practicality and real-time performance. Applicable to large-scale long-distance scenarios such as mountains, it provides reliable support for multi-domain, multi-modal data collaborative applications.

[0012] In one optional implementation, obtaining the initial transformation matrix between the radar coordinate system and the photoelectric sensor coordinate system includes:

[0013] If the installation method of the photoelectric sensor is known and the accuracy meets the preset threshold, long-distance spatial registration is performed directly; if the installation method is unknown or the installation accuracy is not guaranteed, pre-registration within a preset short-range area is performed, which includes:

[0014] Targets are deployed within a preset close range, and target data is collected using radar and photoelectric sensors respectively.

[0015] Extract the coordinates of the target in radar point cloud and photoelectric image;

[0016] The initial transformation matrix is ​​calculated using a pre-defined classic registration algorithm.

[0017] In the embodiments of the present invention, flexible processing is performed according to the sensor installation state. When the installation method is known and the accuracy meets the standard, long-distance registration is directly performed to simplify the process. When the installation information is insufficient, an initial relationship is established through short-distance pre-registration, which not only adapts to different installation conditions and improves flexibility and applicability, but also only arranges short-distance targets when necessary, reducing the scale of target use and deployment difficulty, lowering the implementation cost, and taking advantage of the high data resolution and clear features of short-distance data, combined with classical registration algorithms to ensure the reliability of the initial transformation matrix, providing a high-precision initial benchmark for subsequent long-distance registration, and improving the stability of the overall registration process.

[0018] In an optional implementation manner, calculating the visual consistency error based on the optoelectronic image feature information and calculating the elevation consistency error by using the digital elevation map to match the elevation information of the radar point cloud includes:

[0019] According to the preset elevation demarcation threshold z0 and the elevation information z of the target scene, the scene area where z≥z0 is divided into the upper area, and the scene area where z<z0 is divided into the lower area;

[0020] For the upper area, project the radar point cloud onto the optoelectronic image plane based on the initial transformation matrix, match it with the optoelectronic image feature information, and calculate the distance error between the projected point and the optoelectronic image feature information as the visual consistency error;

[0021] For the lower area, use the digital elevation map to match the elevation information of the radar point cloud, and calculate the elevation difference between the radar point cloud and the elevation information as the elevation consistency error.

[0022] In the embodiments of the present invention, the upper area relies on the matching of the radar point cloud projection and the optoelectronic image features to calculate the visual error, making full use of the advantages of good coverage and clear corresponding relationship between the optoelectronic image and the radar point cloud in this area, improving the utilization rate of sparse point clouds by focusing on the key area, and effectively alleviating the feature matching problem caused by blurred image texture and sparse point clouds at long distances; the lower area uses the digital elevation map to match the elevation information to calculate the elevation error, compensating for the defect that modal data in this area is prone to be missing or contain noise, and ensuring the consistency of terrain geometric features. This partitioned governance strategy takes into account the characteristics of data in different areas, not only giving play to the high-precision advantage of visual features in the upper area, but also enhancing the registration robustness by using the terrain features in the lower area. At the same time, through a clear elevation division standard, the pertinence and effectiveness of error calculation are achieved, providing reliable error input for jointly optimizing the objective function, and significantly improving the accuracy and stability of registration in long-distance scenarios.

[0023] In an optional implementation manner, preprocessing the target image collected by the optoelectronic sensor within a preset long-distance range to generate optoelectronic image feature information includes:

[0024] Semantic segmentation is performed on the target image acquired by the photoelectric sensor within a preset long-distance range to obtain the target mask;

[0025] The process of projecting the radar point cloud onto the photoelectric image plane based on the initial transformation matrix, matching it with photoelectric image feature information, and calculating the distance error between the projected point and the photoelectric image feature information includes:

[0026] Based on the initial transformation matrix, the radar point cloud is projected onto the photoelectric image plane, matched with the edge of the target mask, and the distance error between the projected point and the edge of the target mask is calculated through mask coverage verification.

[0027] This invention provides a target mask by semantically segmenting a long-distance photoelectric image. This allows for precise focusing on key areas (such as large targets like mountains), effectively filtering irrelevant background information and reducing noise interference. Projecting the radar point cloud onto the image plane and matching it with the target mask edges, along with mask coverage verification, fully utilizes the stability of target contours (such as ridgelines) in long-distance scenes. Furthermore, edge distance error calculation quantifies the alignment accuracy between the point cloud and image features, effectively mitigating the problem of insufficient feature matching reliability caused by target scale reduction and texture blurring at long distances. It also improves the utilization rate of sparse radar point clouds in key areas, providing a stable and accurate error measurement basis for visual constraint registration in the upper region, thereby enhancing the accuracy and robustness of registration in long-distance scenes.

[0028] In one optional implementation, the joint optimization objective function is:

[0029]

[0030] Where N is the total number of points used to optimize the radar point cloud; λ1 and λ2 are global weight coefficients; For visual consistency error, For elevation consistency error, w vis w is the dynamic weighting coefficient for visual consistency error. dem w is the dynamic weighting coefficient for elevation consistency error. vis and w dem A transition function is constructed to achieve a smooth transition of weights near the elevation boundary threshold.

[0031] This invention achieves a smooth transition of weights near the elevation boundary threshold by constructing a joint optimization objective that integrates visual consistency error and elevation consistency error, combined with dynamic weight coefficients and a transition function composed of visual and elevation dynamic weight coefficients. This design fully leverages the high precision of visual features in the upper region (such as target mask edges) while utilizing the geometric constraints of elevation information in the lower region (such as digital elevation map data), avoiding the problem of insufficient adaptability of a single error term in complex long-distance scenarios. Simultaneously, the dynamic weight allocation strategy adaptively adjusts the contribution ratio of visual and elevation constraints based on the point cloud elevation, effectively mitigating feature instability caused by sparse radar point clouds and blurred image textures at long distances.

[0032] In one optional implementation, the transition function is expressed as:

[0033]

[0034] w dem (z)=1-w vis (z)

[0035] Where k represents the control transition slope.

[0036] This invention, by introducing a parameter k to control the transition slope, achieves a smooth transition of the weights of visual consistency error and elevation consistency error near the elevation boundary threshold, avoiding the discontinuity in registration results caused by hard partitioning. This smooth transition mechanism can adaptively adjust the contribution ratio of the two errors in joint optimization based on the point cloud elevation values, allowing the upper region to naturally transition to the registration strategy of the lower region. This ensures both the accuracy of registration dominated by visual features in the upper region and the reliability of elevation information constraints in the lower region, effectively improving the overall stability and robustness of long-distance registration in complex terrain environments. It also provides a reasonable and continuous weight allocation basis for solving the joint optimization objective function.

[0037] In a second aspect, the present invention provides a long-range spatial registration system for radar photoelectric sensors, the system comprising:

[0038] The initial transformation matrix acquisition module is used to obtain the initial transformation matrix between the radar coordinate system and the photoelectric sensor coordinate system;

[0039] The photoelectric image feature extraction module is used to preprocess the target image acquired by the photoelectric sensor within a preset long distance range to generate photoelectric image feature information;

[0040] The error calculation module is used to map the radar point cloud to the photoelectric image plane based on the initial transformation matrix, calculate the visual consistency error based on the photoelectric image feature information, and calculate the elevation consistency error by matching the elevation information of the radar point cloud with the digital elevation map.

[0041] The long-range conversion parameter output module is used to construct a joint optimization objective function that includes a dynamic weight allocation of visual consistency error terms and elevation consistency error terms. The optimized conversion parameters are solved by minimizing the joint optimization objective function, thereby realizing long-range spatial registration of radar photoelectric sensors.

[0042] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the long-range spatial registration method of the radar photoelectric sensor described in the first aspect or any corresponding embodiment thereof.

[0043] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the long-range spatial registration method for a radar photoelectric sensor according to the first aspect or any corresponding embodiment described above.

[0044] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the long-range spatial registration method for a radar photoelectric sensor according to the first aspect or any corresponding embodiment described above. Attached Figure Description

[0045] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating a long-range spatial registration method for a radar photoelectric sensor according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the result of directly converting unregistered data into pixel coordinates according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the result of direct conversion of pixel coordinates after close-range registration according to an embodiment of the present invention;

[0049] Figure 4This is a schematic diagram of the pixel coordinate transformation result provided by the long-distance spatial registration method according to an embodiment of the present invention;

[0050] Figure 5 This is a structural block diagram of a long-range spatial registration system for a radar photoelectric sensor according to an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] While existing methods can achieve radar-optical image registration, in long-range detection scenarios, these methods face significant cost increases and accuracy degradation issues as the target distance increases.

[0054] Specific existing methods include:

[0055] 1. Target-based methods: In radar-camera extrinsic parameter registration based on joint targets (such as checkerboard patterns or cubes) or known scene structures (such as building corners), although these traditional methods can provide reliable registration results, their application scenarios have obvious limitations. Especially in long-range registration scenarios, due to the spatial scale limitations and environmental constraints of target placement, it is often difficult to deploy registration reference objects (corner mirrors, checkerboard patterns, etc.) that meet the registration accuracy requirements, which is not conducive to accurate long-range registration of sensors.

[0056] 2. Feature-matching based methods: In radar-camera extrinsic registration methods based on feature matching, it is usually necessary to extract stable feature points from point clouds and images, and optimize the extrinsic matrix by establishing correspondences between features. However, due to the inherent sparsity of radar point cloud data, it is difficult to correspond point cloud features with dense image features. Especially in long-distance scenes, the point cloud resolution is significantly reduced, resulting in a limited number of extractable features and uneven distribution. At the same time, optical images at long distances may also have their feature detection robustness affected by factors such as target size reduction and texture blurring.

[0057] 3. Deep Learning-Based Methods: These methods utilize end-to-end networks to directly learn spatial transformation relationships from raw point cloud and image data. They model sensor data correlations using convolutional neural networks, graph neural networks, or cross-modal attention mechanisms to achieve automatic regression of extrinsic parameter matrices. However, these methods have significant limitations: First, their performance heavily relies on a large amount of precisely labeled multimodal training data, and data acquisition and labeling are costly and time-consuming. Second, in scenarios not covered by training data, the model often exhibits poor generalization ability, leading to a significant decrease in registration accuracy. Furthermore, complex network structures typically require high computational resources, potentially affecting the system's real-time efficiency and making it difficult to meet the real-time requirements of practical applications.

[0058] This embodiment provides a long-range spatial registration method for radar photoelectric sensors to address the shortcomings of the prior art. Figure 1 This is a flowchart of a long-range spatial registration method for a radar photoelectric sensor according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0059] Step S1: Obtain the initial transformation matrix between the radar coordinate system and the photoelectric sensor coordinate system.

[0060] Specifically, the essence of registering radar and photoelectric sensors is to solve for the transformation matrix between the two modal coordinate systems, obtaining the transformation matrix [R|t] from the radar coordinate system to the camera coordinate system:

[0061] P camera =R·P radar +t

[0062] R: 3×3 rotation matrix [θ] x ,θ y ,θ z ], representing the rotation attitude of the radar coordinate system relative to the camera coordinate system in space, and the rotation angles around the three axes X, Y, and Z;

[0063] t: 3×1 translation vector [t x ,t y ,t z ], representing the position offset of the radar coordinate system relative to the camera coordinate system in space, and the spatial distance along the three directions X, Y, and Z.

[0064] The final transformation matrix T has the following form:

[0065] in

[0066] In this embodiment of the invention, if the installation method of the photoelectric sensor is known and the accuracy meets a preset threshold, long-distance spatial registration is performed directly; if the installation method is unknown or the installation accuracy is not guaranteed, pre-registration within a preset short-range area is performed. The process of performing pre-registration within the preset short-range area includes:

[0067] S11, targets are deployed within a preset close range, and target data is collected by radar and photoelectric sensors respectively;

[0068] S12, extract the coordinates of the target in the radar point cloud and photoelectric image;

[0069] S13, calculate the initial transformation matrix using the preset classic registration algorithm.

[0070] Within a preset short-range range (several hundred meters, for example, within 500 meters of the collection point) in this embodiment of the invention. In one specific embodiment, for example, a drone is set up to fly a target and hover at a distance of about 400 meters in different orientations. Four hovering points are marked in the image coordinate system by camera ranging, and four hovering target points are marked in the point cloud by radar point cloud data analysis. After obtaining the coordinates of the same batch of target points in two different coordinate systems, a classical registration algorithm is used to obtain the initial transformation matrix for short range. For example, ICP (Iterative Closest Point) is used, which is a registration algorithm based on point cloud alignment. Iterative optimization minimizes the distance error between the two sets of point clouds. Alternatively, the Kabsch algorithm can be used, which is a rigid body transformation solution method based on SVD, suitable for situations where the correspondence between points is known (such as target point matching). This is only an example and is not a limitation.

[0071] This invention proposes a long-range (hundreds to thousands of meters, e.g., 500 to 5000 meters) registration method that does not rely on target size. It establishes an initial registration relationship through near-range pre-registration and performs long-range extended registration based on this result. When the installation accuracy of radar and photoelectric sensors is high, the target can be completely eliminated, further simplifying the registration process, reducing costs, and improving deployment flexibility. Compared to existing short-range registration schemes (within a few hundred meters), this invention overcomes distance limitations and is suitable for large-scale, long-range collaborative registration of radar and photoelectric sensors.

[0072] Step S2: Preprocess the target image acquired by the photoelectric sensor within a preset long-distance range to generate photoelectric image feature information.

[0073] Specifically, in one embodiment, semantic segmentation is first performed on the target image collected by the optoelectronic sensor. For example, for mountain targets within a distance of 5 kilometers, image segmentation is performed to obtain a mask, and the mask is matched and registered with the initially registered radar point cloud to calculate a new transformation matrix, completing the long-distance registration of the radar point cloud and the optoelectronic image. Using the external camera parameter matrix obtained by registration and the internal camera parameter matrix calculated based on the camera parameters, the image after registration of the radar point cloud and the optoelectronic image can be obtained. The target mountain is more than 1 kilometer away from the collection location. Based on this, redundant point clouds are filtered for registration. Without registration, it is directly converted into pixel coordinate results, such as Figure 2 shown. The pixel coordinate results directly converted after short-distance registration are as shown in Figure 3 shown. Due to the simple installation method of the initial state of the equipment, the normal directions of the two devices are the same and are installed vertically. As much as possible, they are installed at the same position (latitude and longitude). Therefore, the results after simple registration are not very different.

[0074] Step S3: Map the radar point cloud to the optoelectronic image plane based on the initial transformation matrix, calculate the visual consistency error based on the feature information of the optoelectronic image, and calculate the elevation consistency error by using the digital elevation map to match the elevation information of the radar point cloud.

[0075] The embodiment of the present invention takes into account the performance characteristic differences of the data of the two sensors in different height regions and adopts a differential registration strategy: for the upper regions such as the ridge line, since the optoelectronic image and the radar point cloud can usually achieve good coverage and the corresponding relationship is clear, accurate registration is mainly carried out relying on visual consistency constraints, and the alignment accuracy is ensured through feature point matching and mask coverage verification; while for the lower regions such as the foot of the mountain, in view of the situation that modal data is prone to missing or noise, an elevation map is introduced as an auxiliary constraint, and the registration robustness is improved by establishing an error compensation mechanism and a multi-scale fusion algorithm.

[0076] Specifically, the embodiment of the present invention divides the scene area with z≥z0 into the upper region and the scene area with z<z0 into the lower region according to the preset elevation division threshold z0 and the elevation information z of the target scene, and calculates the error terms for different regions respectively:

[0077] 1. For the upper region, project the radar point cloud to the optoelectronic image plane based on the initial transformation matrix, match it with the feature information of the optoelectronic image, and calculate the distance error between the projection point and the feature information of the optoelectronic image as the visual consistency error; specifically, project the radar point cloud to the optoelectronic image plane based on the initial transformation matrix, match it with the edge of the target mask and verify through mask coverage, and calculate the distance error between the projection point and the edge of the target mask.

[0078] 2. For the lower region, use digital elevation maps to match the elevation information of radar point clouds, calculate the elevation difference between radar point clouds and elevation information, and use it as the elevation consistency error.

[0079] Specifically, a Digital Elevation Model (DEM) is a three-dimensional geographic data model that records surface elevation information digitally. It accurately represents the altitude of different locations using regular grids or discrete points and is widely used in Geographic Information Systems (GIS), remote sensing monitoring, and terrain analysis. In its implementation, this invention fully considers the resolution limitations and real-time performance differences of elevation maps. Since elevation maps typically have a resolution of several meters and are prone to undersampling in high-gradient areas, a dynamic weight allocation strategy is adopted to adaptively adjust the contribution ratio of visual constraints to elevation constraints based on regional height, data reliability, and terrain features. This zonal optimization design retains the high-precision advantage of visual data in the upper region while effectively compensating for registration uncertainties in the lower region using elevation information, ultimately achieving stable and reliable long-range radar-electro-optical registration across the entire scene.

[0080] Step S4: Construct a joint optimization objective function that includes the dynamic weight allocation of visual consistency error term and elevation consistency error term, and solve the optimized transformation parameters by minimizing the joint optimization objective function to achieve long-range spatial registration of radar photoelectric sensors.

[0081] Specifically, the joint optimization objective function is:

[0082]

[0083] Where N is the total number of points used to optimize the radar point cloud; λ1 and λ2 are global weight coefficients (used to correct for measurement units, orders of magnitude, etc.); For visual consistency error, For elevation consistency error, w vis w is the dynamic weighting coefficient for visual consistency error. dem w is the dynamic weighting coefficient for elevation consistency error. vis and w dem A transition function is constructed to achieve a smooth transition of weights near the elevation boundary threshold. The transition function is expressed as:

[0084]

[0085] w dem (z)=1-w vis (z)

[0086] Where k represents the control transition slope.

[0087] This invention achieves accurate registration by constructing a joint optimization objective function to simultaneously minimize visual consistency error and elevation consistency error. In the visual error term, the matching error between the radar point cloud and the target mask (such as a mountain outline) edge is calculated by projecting the radar point cloud onto the photoelectric image plane, effectively mitigating the effects of target scale reduction and texture blurring. Intelligent focusing on key areas improves the utilization rate of sparse point clouds. In the elevation error term, the elevation values ​​of the radar point cloud are directly compared with the DEM terrain data, ensuring terrain consistency by minimizing the elevation error between the radar point cloud and the DEM.

[0088] This invention divides the scene into upper and lower regions using an elevation threshold, and employs optimal registration schemes for each. The upper region primarily relies on the visual features of the photoelectric image, achieving registration through mask edge alignment; the lower region prioritizes terrain geometric features to ensure that the point cloud and DEM elevations are consistent. A transition function is designed to achieve a smooth transition between the partition boundaries, avoiding registration discontinuities. This partition-and-conquer strategy effectively solves the problem of single registration methods struggling to balance visual and geometric consistency in large scenes, significantly improving registration robustness in complex terrain environments.

[0089] The pixel coordinate results are transformed using the long-distance spatial registration method provided in this embodiment of the invention, such as... Figure 4 As shown, directly applying the transformation matrix obtained after close-range registration to long-range applications will cause certain errors. However, after long-range registration using the method provided in this embodiment, it performs well in long-range (kilometer-level) radar point cloud photoelectric image registration, with the point cloud distribution better conforming to the terrain contour and improved edge detail consistency. Specifically, it can be seen that before long-range registration... Figure 3 The red dots along the mountain edges are scattered, some overflowing or deviating from the mountain contours; after long-distance registration Figure 4 The red dots along the edges of the mountain more closely follow the mountain's lines, such as at the summit and slope transitions. The point cloud alignment with the mountain's edges (e.g., the outline of the green mountain) is higher, indicating more accurate radar point cloud projection after registration, better matching the actual edge morphology of the mountain in the photoelectric image. The edges where the mountain meets the sky and the vegetation below are also closely aligned. Figure 3 The distribution of midpoint clouds is disordered. Figure 4 The point cloud is arranged in an orderly manner according to the edge contour, which matches better with the mask edge after semantic segmentation of the photoelectric image (such as the boundary of the green area of ​​the mountain). The visual consistency error is reduced, which shows that the registration method makes the point cloud more consistent with the image features in the edge area and enhances the fusion accuracy of multimodal data at the edge.

[0090] This embodiment also provides a long-range spatial registration system for a radar photoelectric sensor, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0091] This embodiment provides a long-range spatial registration system for radar photoelectric sensors, such as... Figure 5 As shown, it includes:

[0092] The initial transformation matrix acquisition module 51 is used to acquire the initial transformation matrix between the radar coordinate system and the photoelectric sensor coordinate system;

[0093] The photoelectric image feature extraction module 52 is used to preprocess the target image acquired by the photoelectric sensor within a preset long distance range to generate photoelectric image feature information;

[0094] Error calculation module 53 is used to map the radar point cloud to the photoelectric image plane based on the initial transformation matrix, calculate the visual consistency error based on the photoelectric image feature information, and calculate the elevation consistency error by matching the elevation information of the radar point cloud with the digital elevation map.

[0095] The long-range conversion parameter output module 54 is used to construct a joint optimization objective function that includes the dynamic weight allocation of visual consistency error term and elevation consistency error term, and solves the optimized conversion parameters by minimizing the joint optimization objective function to achieve long-range spatial registration of radar photoelectric sensors.

[0096] In some optional implementations, the initial transformation matrix acquisition module 51 includes:

[0097] If the installation method of the photoelectric sensor is known and the accuracy meets the preset threshold, long-distance spatial registration is performed directly; if the installation method is unknown or the installation accuracy is not guaranteed, pre-registration within a preset short-distance range is performed.

[0098] The pre-registration within a preset near-range includes:

[0099] Targets are deployed within a preset close range, and target data is collected using radar and photoelectric sensors respectively.

[0100] Extract the coordinates of the target in radar point cloud and photoelectric image;

[0101] The initial transformation matrix is ​​calculated using a pre-defined classic registration algorithm.

[0102] In some optional implementations, the error calculation module 53 includes:

[0103] A region division unit, configured to divide a scene region where z≥z0 into an upper region and a scene region where z<z0 into a lower region according to a preset elevation boundary threshold z0 and the elevation information z of the target scene;

[0104] A visual error calculation unit, configured to project the radar point cloud onto the optoelectronic image plane based on the initial transformation matrix for the upper region, match it with the optoelectronic image feature information, and calculate the distance error between the projected point and the optoelectronic image feature information as the visual consistency error;

[0105] An elevation error calculation unit, configured to use the digital elevation map to match the elevation information of the radar point cloud for the lower region and calculate the elevation difference between the radar point cloud and the elevation information as the elevation consistency error.

[0106] In some optional embodiments, the optoelectronic image feature extraction module 52 includes: performing semantic segmentation on the target image collected by the optoelectronic sensor within a preset long-distance range to obtain a target mask;

[0107] The visual error calculation unit projects the radar point cloud onto the optoelectronic image plane based on the initial transformation matrix, matches it with the edge of the target mask, and calculates the distance error between the projected point and the edge of the target mask through mask coverage verification.

[0108] In some optional embodiments, the joint optimization objective function in the long-distance conversion parameter output module 54 is:

[0109]

[0110] where N is the total number of points used to optimize the radar point cloud; λ1 and λ2 are global weight coefficients; is the visual consistency error, is the elevation consistency error, w vis is the dynamic weight coefficient of the visual consistency error, w dem is the dynamic weight coefficient of the elevation consistency error, w vis and w dem constitute a transition function to achieve smooth transition of the weight near the elevation boundary threshold.

[0111] In an optional embodiment, the transition function is expressed as:

[0112]

[0113] w dem (z)=1 - w vis (z)

[0114] where k represents the control transition slope.

[0115] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0116] In this embodiment, the long-range spatial registration system of the radar photoelectric sensor is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0117] This invention also provides a computer device having the above-described features. Figure 5 The diagram shows a long-range spatial registration system for a radar photoelectric sensor.

[0118] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0119] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0120] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0121] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0123] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0124] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0125] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0126] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for long-range spatial registration of a radar photoelectric sensor, characterized in that, Including: Obtain an initial transformation matrix between the radar coordinate system and the optoelectronic sensor coordinate system; Preprocess the target images collected by the optoelectronic sensor within a preset long-distance range to generate optoelectronic image feature information; Map the radar point cloud to the optoelectronic image plane based on the initial transformation matrix, calculate the visual consistency error based on the optoelectronic image feature information, and calculate the elevation consistency error by using the digital elevation map to match the elevation information of the radar point cloud; Construct a joint optimization objective function including dynamic weight allocation of the visual consistency error term and the elevation consistency error term, and solve the optimized transformation parameters by minimizing the joint optimization objective function to achieve long-distance spatial registration of the radar optoelectronic sensor.

2. The method according to claim 1, characterized in that, The obtaining of the initial transformation matrix between the radar coordinate system and the optoelectronic sensor coordinate system includes: If the installation method of the optoelectronic sensor is known and the accuracy meets the preset threshold, directly perform long-distance spatial registration; if the installation method is unknown or the installation accuracy is not guaranteed, perform pre-registration within a preset short-distance range. The performing of the pre-registration within a preset short-distance range includes: Deploy a target within a preset short-distance range, and collect target data through the radar and the optoelectronic sensor respectively; Extract the coordinates of the target in the radar point cloud and the optoelectronic image; Use a preset classical registration algorithm to calculate the initial transformation matrix.

3. The method according to claim 1, characterized in that, The calculating of the visual consistency error based on the optoelectronic image feature information and the calculating of the elevation consistency error by using the digital elevation map to match the elevation information of the radar point cloud include: According to a preset elevation demarcation threshold z0 and the elevation information z of the target scene, divide the scene area where z≥z0 into the upper area, and divide the scene area where z<z0 into the lower area; For the upper area, project the radar point cloud to the optoelectronic image plane based on the initial transformation matrix, match it with the optoelectronic image feature information, and calculate the distance error between the projection point and the optoelectronic image feature information as the visual consistency error; For the lower area, use the digital elevation map to match the elevation information of the radar point cloud, and calculate the elevation difference between the radar point cloud and the elevation information as the elevation consistency error.

4. The method according to claim 3, characterized in that, The preprocessing of the target images collected by the optoelectronic sensor within a preset long-distance range to generate optoelectronic image feature information includes: Perform semantic segmentation on the target images collected by the optoelectronic sensor within a preset long-distance range to obtain a target mask; The projecting of the radar point cloud to the optoelectronic image plane based on the initial transformation matrix, matching with the optoelectronic image feature information, and calculating the distance error between the projection point and the optoelectronic image feature information includes: Project the radar point cloud to the optoelectronic image plane based on the initial transformation matrix, match it with the edge of the target mask, and calculate the distance error between the projection point and the edge of the target mask through mask coverage verification.

5. The method according to claim 3, characterized in that, The joint optimization objective function is: Where N is the total number of points used to optimize the radar point cloud; λ1 and λ2 are global weight coefficients; For visual consistency error, For elevation consistency error, w vis w is the dynamic weighting coefficient for visual consistency error. dem w is the dynamic weighting coefficient for elevation consistency error. vis and w dem A transition function is constructed to achieve a smooth transition of weights near the elevation boundary threshold.

6. The method according to claim 5, characterized in that, The transition function is expressed as: In dem (with)=1-in vis (With) where k represents the control transition slope.

7. A long-range spatial registration system for a radar photoelectric sensor, characterized in that, Including: An initial transformation matrix acquisition module for obtaining an initial transformation matrix between the radar coordinate system and the optoelectronic sensor coordinate system; An optoelectronic image feature extraction module for preprocessing the target images collected by the optoelectronic sensor within a preset long-distance range to generate optoelectronic image feature information; The error calculation module is used to map the radar point cloud to the photoelectric image plane based on the initial transformation matrix, calculate the visual consistency error based on the photoelectric image feature information, and calculate the elevation consistency error by matching the elevation information of the radar point cloud with the digital elevation map. The long-range conversion parameter output module is used to construct a joint optimization objective function that includes a dynamic weight allocation of visual consistency error terms and elevation consistency error terms. The optimized conversion parameters are solved by minimizing the joint optimization objective function, thereby realizing long-range spatial registration of radar photoelectric sensors.

8. A computer device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the long-range spatial registration method for the radar photoelectric sensor according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the long-range spatial registration method of the radar photoelectric sensor according to any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions for causing a computer to execute the long-range spatial registration method for the radar photoelectric sensor as described in any one of claims 1 to 6.

Images