Method, system and terminal for automatic registration of radar-visual space based on image feature learning
Through the method based on image feature learning, the problem of low registration accuracy of radar and visual data is solved, and the precise registration of radar and visual data is achieved, which reduces the cost and difficulty of system maintenance and improves target detection and tracking in intelligent traffic ability.
Patent Information
- Application Number
- CN202410889922.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-04
AI Technical Summary
The existing automatic registration method based on targetless is difficult to ensure the precise registration of radar detection targets and visual detection targets, and there are problems of accuracy and cost.
Using the automatic registration method of Lei Vision space based on image feature learning, the rigid body transformation matrix is initialized by collecting radar point cloud data and visual camera images, and the projected image is corrected based on similarity, and the rigid body transformation matrix is optimized to realize the point cloud data and plane image. Automatic registration.
It realizes accurate registration of radar and visual data, reduces system maintenance costs and difficulty, and improves the accuracy of target detection and tracking.
Smart Images

Figure CN118429402B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a radar-visual space automatic registration method, system and terminal based on image feature learning. Background Art
[0002] In intelligent transportation systems, the application of millimeter-wave radar and camera fusion technology is one of the key technologies for achieving accurate perception and safe decision-making. The registration technology of millimeter-wave radar and camera is a prerequisite for accurate sensor fusion detection. Through accurate data registration, the consistency of radar detection targets and visual detection targets can be ensured, thereby improving the accuracy and tracking ability of target detection. Due to the difference in sensing principles between millimeter-wave radar and camera, millimeter-wave radar usually provides three-dimensional point cloud data, while cameras provide two-dimensional image data. Therefore, the problem of spatial registration of radar and visual data is essentially a cross-modal 2D-3D feature matching solution problem.
[0003] At present, there are two common solution methods. The first is a target-based automatic registration method, which mainly uses a special trihedral reflector to collect radar data of a preset target object, and then uses a camera to collect image data at the same time. The signal coincidence points generated by different modal data are used to solve the conversion correspondence between the radar and the camera in space. The advantage of this method is simple operation, but the disadvantage is that the detection accuracy of the equipment is high, and the registration workload is large and the labor cost is high. The second is a target-free automatic registration method, which does not rely on the preset target. According to the correlation of the synchronously collected data of the millimeter-wave radar and the camera, mathematical modeling is performed to estimate the optimal external parameters, thereby realizing the automatic registration of the radar and the camera in space. Although this method does not require high equipment accuracy and is low in cost, it has high requirements on the environmental conditions of the scene application and the accuracy of the mathematical model. There is often a problem of difficulty in ensuring the accurate registration of the radar detection target and the visual detection target. Summary of the invention
[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method, system and terminal for automatic radar-visual space registration based on image feature learning, aiming to solve the problem in the prior art that it is difficult to ensure accurate registration of radar detection targets and visual detection targets based on targetless automatic registration methods.
[0005] In order to achieve the above object, the present invention provides a first aspect of a radar-visual space automatic registration method based on image feature learning, comprising:
[0006] Collect the point cloud data of the radar and the plane image of the visual camera at the same time;
[0007] Initialize a rigid body transformation matrix between the radar and the visual camera, and based on the rigid body transformation matrix, project the point cloud data into a plane coordinate system where the plane image is located to obtain a projected image;
[0008] Based on the similarity between the projection image and the plane image, the projection image is corrected to obtain an optimized projection image;
[0009] Based on the mapping relationship between the point cloud data and the optimized projection image, an optimized rigid body transformation matrix is obtained;
[0010] Based on the optimized rigid body transformation matrix, the point cloud data and the plane image are automatically registered, and a registration result is output.
[0011] In one implementation, projecting the point cloud data onto the plane coordinate system where the plane image is located based on the rigid body transformation matrix to obtain a projected image includes:
[0012] Obtaining the straight-line distance from the target object to the plane coordinate system to obtain the depth of the projected pixel;
[0013] Acquire the intrinsic parameter matrix of the visual camera, establish a mapping relationship between a single pixel of the point cloud data and the plane coordinate system based on the intrinsic parameter matrix, the depth of the projection pixel and the rigid body transformation matrix, and obtain a projection equation;
[0014] Based on the projection equation, the point cloud data is projected into the plane coordinate system to obtain a projection image.
[0015] In one implementation, the correcting the projection image based on the similarity between the projection image and the plane image to obtain an optimized projection image includes:
[0016] constructing a cost function based on the similarity between the projection image and the plane image;
[0017] Based on the cost function, the projection image is corrected to obtain an optimized similarity;
[0018] The step of correcting the projection image is repeatedly performed until the optimized similarity meets a preset similarity threshold, thereby obtaining an optimized projection image.
[0019] In one implementation, constructing a cost function based on the similarity between the projection image and the plane image includes:
[0020] Performing feature extraction on the projection image and the plane image respectively to obtain a projection feature map and a plane feature map;
[0021] Flattening the projection feature map and the plane feature map to obtain a projection feature vector and a plane feature vector respectively;
[0022] A cost function is constructed based on the similarity between the projection feature vector and the position vector of each pixel in a preset pixel range of the plane feature vector.
[0023] In one implementation, the correcting the projection image based on the cost function to obtain an optimized similarity includes:
[0024] Correcting the projected image within a preset pixel range to obtain a corrected projected image;
[0025] The similarity between the corrected projection image and the plane image represented by the cost function within the pixel range is solved to obtain an optimized similarity.
[0026] In one implementation, obtaining an optimized rigid body transformation matrix based on a mapping relationship between the point cloud data and the optimized projection image includes:
[0027] Based on the projection equation and the depth of the projection pixel, mapping the optimized projection image to a three-dimensional space to obtain reconstructed point cloud data;
[0028] A rigid body transformation is performed based on the reconstructed point cloud data and the projection image to obtain an optimized rigid body transformation matrix.
[0029] In one implementation, after outputting the registration result, the method further includes calculating the registration error, which specifically includes:
[0030] Based on the optimized rigid body transformation matrix, obtaining an extrinsic parameter prediction value and an extrinsic parameter true value;
[0031] Acquire the initial extrinsic parameters of the radar, and construct a point cloud distance loss function based on the point cloud data, the initial extrinsic parameters, the extrinsic parameter prediction value and the extrinsic parameter true value;
[0032] Based on the point cloud distance loss function, a registration error is obtained.
[0033] A second aspect of the present invention provides a radar-visual space automatic registration system based on image feature learning, comprising:
[0034] The information acquisition module is used to collect the point cloud data of the radar and the plane image of the visual camera at the same time;
[0035] A projection module, used for initializing a rigid body transformation matrix between the radar and the visual camera, and projecting the point cloud data into a plane coordinate system where the plane image is located based on the rigid body transformation matrix to obtain a projected image;
[0036] A matching module, used for correcting the projection image based on the similarity between the projection image and the plane image to obtain an optimized projection image;
[0037] A rigid body transformation matrix optimization module, used for obtaining an optimized rigid body transformation matrix based on a mapping relationship between the point cloud data and the optimized projection image;
[0038] A registration module is used to automatically register the point cloud data and the plane image based on the optimized rigid body transformation matrix, and output a registration result.
[0039] In one embodiment, the matching module includes a similarity calculation unit and a matching optimization unit, wherein the similarity calculation unit is used to construct a cost function based on the similarity between the projection image and the plane image; the matching optimization unit is used to iteratively correct the projection image until the similarity of the cost function meets a preset similarity threshold, thereby obtaining an optimized projection image.
[0040] A third aspect of the present invention provides a terminal, comprising a memory, a processor, and a radar-visual space automatic registration program based on image feature learning stored in the memory and executable on the processor, wherein the radar-visual space automatic registration program based on image feature learning, when executed by the processor, implements any one of the steps of the above-mentioned radar-visual space automatic registration method based on image feature learning.
[0041] A fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a program for automatic radar spatial registration based on image feature learning. When the program for automatic radar spatial registration based on image feature learning is executed by a processor, the program implements any step of the above-mentioned method for automatic radar spatial registration based on image feature learning.
[0042] Compared with the prior art, the beneficial effects of this solution are as follows:
[0043] The present invention collects the point cloud data of the radar and the plane image of the visual camera at the same time; initializes the rigid body transformation matrix between the radar and the visual camera, and based on the rigid body transformation matrix, projects the point cloud data into the plane coordinate system where the plane image is located to obtain a projection image; based on the similarity between the projection image and the plane image, corrects the projection image to obtain an optimized projection image; based on the mapping relationship between the point cloud data and the optimized projection image, obtains the optimized rigid body transformation matrix; based on the optimized rigid body transformation matrix, automatically aligns the point cloud data and the plane image, and outputs the alignment result. The present invention initializes a rigid body transformation matrix to perform initial matching between the point cloud data and the plane image, and optimizes the rigid body transformation matrix through feedback self-learning through the radar-visual space automatic alignment network, thereby automatically aligning the point cloud data and the plane image, and outputting an accurate alignment result. The model structure is simple in design and can effectively reduce the system maintenance cost and difficulty. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0045] Figure 1 It is a flow chart of the automatic registration method of the radar-visual space based on image feature learning of the present invention;
[0046] Figure 2 It is a framework diagram of the automatic registration method of the radar-visual space based on image feature learning of the present invention;
[0047] Figure 3 A schematic diagram of the projection relationship of the radar point cloud data of the present invention onto the plane where the visual camera is located;
[0048] Figure 4 A comparison diagram before and after registration based on the first initial external parameter of the present invention;
[0049] Figure 5 A comparison diagram before and after the registration based on the second initial external parameter of the present invention;
[0050] Figure 6 It is a schematic diagram of the module of the radar-visual space automatic registration system based on image feature learning of the present invention;
[0051] Figure 7 It is a schematic diagram of the terminal structure of the present invention. DETAILED DESCRIPTION
[0052] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.
[0053] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0054] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0055] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0056] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0058] In view of the problem that the existing targetless automatic registration methods are difficult to ensure the accurate registration of radar detection targets and visual detection targets, the present invention proposes an automatic registration method for radar-visual space based on image feature learning. The method mainly takes deep learning as theoretical knowledge and constructs a mathematical model with strong generalization ability through the self-learning ability of a large amount of scene data, reduces the constraints of environmental conditions on mathematical modeling, and reduces the difficulty and cost of manual maintenance.
[0059] The embodiment of the present invention provides a radar-visual space automatic registration method based on image feature learning, which is deployed on electronic devices such as computers and servers. The application scenario is target detection and tracking in intelligent transportation. It is mainly achieved by registering point cloud data collected by radar (such as millimeter wave radar, etc.) and plane image data collected by visual cameras (such as digital cameras or video cameras, etc.) through a radar-visual space automatic registration network. Specifically, Figure 1 and Figure 2 As shown, the radar-visual space automatic registration network of the present invention mainly consists of three parts: feature extraction layer, feature matching layer and three-dimensional space conversion layer. The steps of the method of this embodiment include:
[0060] Step S100: collecting the point cloud data of the radar and the plane image of the visual camera at the same time;
[0061] Specifically, since the premise of achieving registration is that the point cloud data and the plane image data need to correspond to the data of the same target area collected at the same time and under the same environmental conditions, in order to ensure that the two data are collected at the same time, this embodiment uses multi-sensor data fusion technology and timestamp calibration technology to ensure that the millimeter wave radar and the visual camera collect data at the same time. The millimeter wave radar is used to read the point cloud data from the echo signal. Each point contains a three-dimensional coordinate information, and also contains information such as color, reflection intensity, distance from the millimeter wave radar to the target, position and target speed; the visual camera is used to capture the plane image of the target area by scanning line by line or frame by frame.
[0062] Step S200: Initializing a rigid body transformation matrix between the radar and the visual camera, and based on the rigid body transformation matrix, projecting the point cloud data into a plane coordinate system where the plane image is located to obtain a projected image;
[0063] Before initializing the rigid body transformation matrix, the radar and visual camera need to be calibrated to determine the relative position and attitude between the two. Then, a rigid body transformation matrix is initialized based on the position and attitude between the two to determine the rotation and translation relationship between the radar coordinate system and the visual camera coordinate system. The rigid body transformation matrix is used to construct the projection equation to project the point cloud data from the three-dimensional coordinate system to the plane coordinate system where the plane image data is located. Specifically, each pixel point in the radar coordinate system is first converted to the visual camera coordinate system through the rigid body transformation matrix; then it is converted to a three-dimensional pixel point in the visual camera coordinate system, and projected to the two-dimensional image coordinate system where the two-dimensional image plane is located through the intrinsic parameter matrix (including focal length, principal point coordinates, etc.), that is, the radar point cloud data is represented in the visual camera plane image coordinate system. It is worth stating that the two-dimensional coordinates of the three-dimensional pixel points projected onto the image plane may not be integer coordinates. For ease of processing, they can also be converted to integer coordinates through operations such as interpolation or rounding.
[0064] Step S300: based on the similarity between the projection image and the plane image, the projection image is corrected to obtain an optimized projection image;
[0065] Specifically, the projected image is corrected based on the similarity between the projected image and the visual camera plane image, which is an iterative optimization process, aiming to improve the alignment accuracy of the projected image and the plane image. Based on the principle of optical flow estimation, this embodiment regards the projected image and the plane image as adjacent video frames, and evaluates the similarity between the projected image and the plane image by calculating the similarity between each position vector of the projected image and the plane image.
[0066] First, based on the rigid body transformation matrix obtained by initialization, the radar point cloud data is projected into the coordinate system where the visual camera plane image is located to obtain the initial projection image. The feature extraction layer extracts features from the initial projection image and the plane image respectively to obtain the projection feature map and the plane feature map. The feature matching layer matches the projection feature map with the plane feature map according to the pixel position, and the matching results are evaluated for similarity to calculate a similarity score. Then, based on the similarity score, the point cloud distance is used as the loss function to iteratively adjust the parameters in the rigid body transformation matrix so that the similarity score between the projection image and the plane image meets the preset similarity threshold or reaches the preset number of iterative optimizations to obtain the optimized projection image. It can be seen that the optimized projection image and the plane image have achieved a high degree of matching in similarity, which lays a good foundation for subsequent tasks such as target detection or tracking.
[0067] Step S400: obtaining an optimized rigid body transformation matrix based on a mapping relationship between the point cloud data and the optimized projection image;
[0068] Specifically, after the optimized projection image is obtained, the constructed projection equation is used to describe the mapping relationship between the point cloud data and the optimized projection image, thereby obtaining the optimized rigid body transformation matrix.
[0069] Step S500: Based on the optimized rigid body transformation matrix, the point cloud data and the plane image are automatically registered, and a registration result is output.
[0070] Specifically, the optimized rigid body transformation matrix is used to transform the point cloud data from the radar coordinate system to the plane coordinate system where the plane image is located, so as to achieve automatic alignment of the positions of each pixel point.
[0071] In this embodiment, by initializing a rigid body transformation matrix, the projection image of the point cloud data in the coordinate system where the plane image is located is preliminarily matched with the plane image, and then the position and posture of the initial rigid body transformation matrix are gradually adjusted to adjust the projection image, so as to iteratively optimize the similarity of the feature matching results until the similarity threshold is met, indicating that the optimized rigid body transformation matrix obtained at this time can accurately match the projection image obtained by point cloud data conversion with the plane image, that is, by adjusting the relative position and posture relationship between the radar and the visual camera, the point cloud data and the plane image can be automatically aligned. In this embodiment, in the process of optimizing the rigid body transformation matrix, the radar-visual space automatic alignment network can perform feedback self-learning, so as to obtain a trained radar-visual space automatic alignment network, and then the trained radar-visual space automatic alignment network can be used to automatically align the point cloud data and the plane image, and output an accurate alignment result. The trained radar-visual space automatic alignment network is not constrained by environmental conditions, has good generalization ability, can quickly realize target detection or tracking, and has a simple model structure design, which can effectively reduce the system maintenance cost and difficulty.
[0072] In one implementation, the step S200 of projecting the point cloud data onto the plane coordinate system where the plane image is located based on the rigid body transformation matrix to obtain a projected image includes:
[0073] Step S210: obtaining the straight-line distance from the target object to the plane coordinate system to obtain the depth of the projection pixel;
[0074] Step S220: acquiring the intrinsic parameter matrix of the visual camera, and establishing a mapping relationship between a single pixel of the point cloud data and the plane coordinate system based on the intrinsic parameter matrix, the depth of the projection pixel and the rigid body transformation matrix, to obtain a projection equation;
[0075] Step S230: Based on the projection equation, project the point cloud data into the plane coordinate system to obtain a projection image.
[0076] Specifically, since the millimeter-wave radar and the camera need to be registered, the transformation relationship between the radar measurement results and the camera image must be correctly estimated, so that the detection results and the corresponding objects in the image can be spatially consistent. Figure 3 As shown, assuming that the radar and the visual camera observe the same target P point, through the initial rotation matrix and the initial translation vector Construct the rigid body transformation matrix between the initialization radar and the visual camera ,Right now:
[0077] (1),
[0078] Then, use the initialized rigid body transformation matrix Construct a projection equation and use it to project the point cloud data from its three-dimensional space to the two-dimensional coordinate system where the plane image is located. The two-dimensional coordinate system here refers to the three-dimensional coordinate system where the visual camera is located. Observation target P The plane coordinate system of the plane image is obtained by shooting at the point, thereby obtaining the projection image, where the projection equation is:
[0079] (2),
[0080] in, are the coordinates corresponding to the two coordinate axes in the plane coordinate system where the plane image taken by the visual camera is located. The three-dimensional coordinate system of the point cloud data captured by the radar The coordinates corresponding to the three coordinate axes below are is the straight-line distance from the object to the camera image plane, that is, the depth of the projected pixel, is the intrinsic parameter matrix of the camera.
[0081] In this embodiment, the point cloud data is projected from its three-dimensional space to the two-dimensional coordinate system where the plane image is located by constructing a projection equation, thereby achieving one-to-one registration of pixel points at corresponding positions in the point cloud data and the plane image in the two-dimensional plane, and during the registration process, the rotation and translation parameters of the projected image can be continuously adjusted to optimize the registration accuracy, that is, the registration accuracy is optimized by continuously adjusting the rigid body transformation matrix, which is simple and efficient.
[0082] In one embodiment, the step S300 of correcting the projection image based on the similarity between the projection image and the plane image to obtain an optimized projection image includes:
[0083] Step S310: Based on the similarity between the projection image and the plane image, constructing a cost function, specifically comprising: performing feature extraction on the projection image and the plane image respectively to obtain a projection feature map and a plane feature map; flattening the projection feature map and the plane feature map to obtain a projection feature vector and a plane feature vector respectively; constructing a cost function based on the similarity between the position vectors of each pixel of the projection feature vector and the plane feature vector within a preset pixel range.
[0084] Specifically, since the coordinate system corresponding to the point cloud data output by the millimeter wave radar device is a Cartesian coordinate system, in order to facilitate the projection of the point cloud data into a two-dimensional plane coordinate system, this embodiment performs coordinate transformation on the point cloud data and converts it into an image coordinate system to generate a sparse depth map.
[0085] Due to the initialized rigid body transformation matrix It is randomly generated according to the initial position and initial posture between the millimeter wave radar and the visual camera. The measurement target represented by the projection image generated by projecting the point cloud data onto the image plane will have a large position deviation from the measurement target of the plane image. Therefore, this embodiment normalizes the sparse depth map so that the depth range is normalized to The normalized sparse depth map is pooled within the range of , and a 5×5 maximum pooling window is used to perform maximum pooling on the normalized sparse depth map to obtain a semi-dense depth map, which is beneficial to reduce the spatial dimension of the depth map while retaining important information, reduce the number of parameters in the network, and thus reduce the computational complexity.
[0086] like Figure 2 As shown in the figure, the feature extraction layer in the Lei Shi spatial automatic registration network uses the ResNet-18 residual neural network to construct two feature extraction branches, namely, the projection feature extraction branch for extracting projection image features and the plane feature extraction branch for extracting plane image features. Among them, the network structure of the plane feature extraction branch is: the ResNet-18 residual neural network is divided into five convolutional network modules, with a total of 18 layers. The first convolutional network module has a 7×7 convolution kernel and 64 channels. The second convolutional network module to the fifth convolutional network module all have 4 3×3 convolutional kernel layers, the second convolutional network module has 64 channels, the third convolutional network module has 128 channels, the fourth convolutional network module has 256 channels, and the fifth module convolution has 512 channels. Based on the structural design of this network, the input data dimension 64×112×112 can be reduced to 512×7×7 and then output, achieving the purpose of dimensionality reduction, and can further screen information with significant features. The network structure of the projection feature extraction branch is also based on the ResNet-18 residual neural network. However, compared with the network structure design of the plane feature extraction branch, the number of filters in the five convolutional network modules is halved, which can significantly improve the computational efficiency. Furthermore, since the millimeter-wave radar and camera are installed on the gantry of the highway or the traffic light pole on the urban road, it is inconvenient to disassemble and assemble. Therefore, the parameters of the convolutional network modules in the two feature extraction branches can be pre-trained in advance to learn the basic feature information of images such as vehicles, pedestrians, guardrails and roads in advance, which is conducive to improving the training efficiency of the entire network model.
[0087] Then, the projection feature map and the plane feature map output by the projection feature extraction branch and the plane feature extraction branch are feature matched along the channel dimension. Specifically, based on the principle of optical flow estimation, the similarity between the position vectors of each pixel of the projection feature vector and the plane feature vector within a preset pixel range is calculated, and a cost function is constructed, which is expressed as follows:
[0088] (3),
[0089] in, It is a feature map Flat vector of is a column vector Length, is the transposition operator, It is customized for dot multiplication reflect and A function that measures the similarity between two vectors.
[0090] Since the feature information extracted by the projection feature extraction branch and the plane feature extraction branch is mainly used for feature matching rather than image classification, the projection feature extraction branch in this embodiment adopts a strategy of halving the number of filters in each stage. This can improve the computational efficiency while ensuring the effectiveness of feature extraction, and the algorithm model can also be directly deployed in embedded hardware devices to achieve fast autonomous iterative learning.
[0091] Step S320: Based on the cost function, the projection image is corrected to obtain an optimized similarity; the step of correcting the projection image is repeated until the optimized similarity satisfies a preset similarity threshold to obtain an optimized projection image, specifically comprising: correcting the projection image within a preset pixel range to obtain a corrected projection image; solving the similarity between the corrected projection image and the plane image represented by the cost function within the pixel range to obtain the optimized similarity; repeating the step of correcting the projection image until the similarity within the preset pixel range satisfies a preset similarity threshold to obtain an optimized projection image.
[0092] If the similarity calculation is performed on the vectors of the projection feature vector and the plane feature vector at all positions, the calculation complexity and cost are relatively high. Therefore, this embodiment adopts the method of calculating the similarity of the projection feature vector and the plane feature vector within a certain pixel range to significantly reduce the calculation complexity, wherein the selection of the pixel range can be set according to the required indicators such as calculation accuracy, model complexity, and calculation amount, and the selected pixel range can ensure that the expected registration accuracy requirements are met. Specifically, the resolution of the original plane feature map is downsampled to 1 / 32 through the five convolutional network modules in the network structure of the plane feature extraction branch. Therefore, it is only necessary to ensure that the projection feature vector and the plane feature vector are within a certain pixel range. The cost within the pixel range is:
[0093] (4),
[0094] in, The closer the cost value is to 1, the higher the similarity between the projection feature vector and the plane feature vector. If the preset similarity threshold is 0.9, the projection image is corrected by adjusting the rigid body transformation matrix to obtain the corrected projection image. The cost function is solved in The similarity between the corrected projection image and the plane image represented within the pixel range is calculated to obtain the optimized similarity; the step of correcting the projection image is repeated until the similarity is calculated based on the last corrected projection image. , it indicates that the similarity satisfies the similarity threshold of 0.9, and the corrected projection image at this time is used as the optimized projection image.
[0095] In this embodiment, based on the principle of optical flow estimation, a feature map similarity calculation method based on matrix product calculation is designed, the plane image remains unchanged, and the projected image is corrected by adjusting the rigid body transformation matrix, thereby optimizing the cost to improve the similarity. Furthermore, in order to improve the calculation efficiency, the overall similarity between the corresponding position vectors of the projection feature vector and the plane feature vector within a preset pixel range is used as the final similarity, and the optimization iterative process has a small amount of calculation and is simple and efficient.
[0096] In one implementation, the step of obtaining an optimized rigid body transformation matrix based on a mapping relationship between the point cloud data and the optimized projection image in step S400 includes:
[0097] Step S410: Based on the projection equation and the depth of the projection pixel, mapping the optimized projection image to a three-dimensional space to obtain reconstructed point cloud data;
[0098] Step S420: performing a rigid body transformation based on the reconstructed point cloud data and the projection image to obtain an optimized rigid body transformation matrix.
[0099] Specifically, the optimized projected image is passed through the fully connected layer network (FC layer) to output a 6×1 vector, in which the first three elements are related to rotation and the last three elements are related to translation. The 3D grid generator is based on the 2D grid generator in the Spatial Transformer Network (STN). It uses the depth map composed of the depth of the projected pixels as an additional input to map the 2D image pixels to the corresponding 3D pixel points in the world coordinate system and use The rigid body transformation motion of the layer transforms these three-dimensional pixels. Layer pair 3×1 rotation matrix Copy the generated matrix and then splice the 3×1 translation vector , and finally generate a 3×4 matrix .
[0100] In this embodiment, the optimized projection image is reversely mapped into the three-dimensional space using the constructed projection equation and the depth of the projection pixel to obtain an optimized rigid body transformation matrix. The rotation parameters and translation parameters corresponding to the rigid body transformation matrix at this time represent the rotation angle and the movement distance required to adjust the millimeter wave radar.
[0101] In one implementation, after outputting the registration result in step S500, step S600 is further included: calculating the registration error, which specifically includes:
[0102] Step S610: obtaining an extrinsic parameter prediction value and an extrinsic parameter true value based on the optimized rigid body transformation matrix;
[0103] Step S620: Acquire the initial external parameters of the radar, and construct a point cloud distance loss function based on the point cloud data, the initial external parameters, the external parameter prediction value and the external parameter true value;
[0104] Step S630: Obtaining a registration error based on the point cloud distance loss function.
[0105] Specifically, assuming that a set of radar point cloud data is given , define the loss function used to represent the registration error , the formula is as follows:
[0106] (5),
[0107] in, is the initial external parameter randomly generated during training data, is the predicted external parameter, is the truth value of the external parameter, , is the true value of the rotation matrix, is the true value of the translation matrix. The external parameter here refers to the rotation parameter between the millimeter wave radar and the visual camera. and translation parameters .if The closer the value is to the identity matrix , then the predicted Point cloud data and the real The smaller the distance deviation between the point cloud data, the Point cloud data The smaller the loss of the projection transformation, the smaller the overall registration error will be if the deviation of the distance of each point cloud is small. In this example, the radar is first calibrated using a corner reflector at the construction site to estimate the point cloud coordinates in the three-dimensional coordinate system where the radar is located. , and then convert it through formula (2) to obtain the true value of the external parameter .
[0108] In the present embodiment, during the training process of the radar visual space automatic registration network, the above-mentioned point cloud distance is used as the loss function. From the definition of the loss function, it can be seen that it is only related to the geometric structure information between the radar point cloud data, but has nothing to do with the intrinsic parameters of the camera, thereby achieving decoupling from the intrinsic parameters of the camera. Obviously, when the intrinsic parameters of the camera change, the radar visual space automatic registration network model may not need to be retrained or only needs to be fine-tuned before it can be put into use.
[0109] A loss function based on point cloud distance calculation designed in this embodiment can significantly reduce or even avoid the impact of camera intrinsic parameter transformation on the radar space automatic registration network model, thereby helping to reduce model training costs and improve registration efficiency and accuracy.
[0110] The effectiveness of the matching method proposed in the present invention is verified by experiments. Under different distances between the millimeter-wave radar and the visual camera, and the angle between the illumination surfaces of the millimeter-wave radar and the visual camera, the rotation angle and the translation distance calculated by the method of the present invention are adjusted. The set initial relative position, parameter adjustment data and alignment error are shown in Table 1.
[0111] Table 1:
[0112]
[0113] As shown in Table 1, the method of the present invention achieves experimental verification results with an average translation error of 2.64 cm (translation errors of the x-axis, y-axis, and z-axis are 3.07 cm, 2.52 cm, and 2.34 cm, respectively) and an average angle error of 0.27° (angle errors in the rolling, pitching, and yaw directions are 0.21°, 0.30°, and 0.29°, respectively). The experimental results show that the smaller the physical displacement between the millimeter-wave radar and the camera, the smaller the registration error of the method of the present invention.
[0114] The initial position and projection position of the point cloud data before automatic registration corresponding to the data in Table 1 are as follows: Figure 4 As shown in the figure, the point cloud data position and projection position after automatic registration are as follows: Figure 5 As shown in the figure, the true value of the point cloud data collected by the radar is marked with the origin, and the projection result is marked with a triangle. Figure 4 and Figure 5It can be seen that there is a large deviation between the initial position and the projection position of the point cloud data before automatic registration, and the position and the projection position of the point cloud data after automatic registration are basically consistent. The robustness of the method of the present invention has been effectively verified.
[0115] like Figure 6 As shown, corresponding to the above-mentioned automatic radar-visual space registration method based on image feature learning, an embodiment of the present invention further provides an automatic radar-visual space registration system based on image feature learning, and the above-mentioned automatic radar-visual space registration system based on image feature learning includes:
[0116] The information acquisition module 610 is used to acquire the point cloud data of the radar and the plane image of the visual camera at the same time;
[0117] A projection module 620 is used to initialize a rigid body transformation matrix between the radar and the visual camera, and based on the rigid body transformation matrix, project the point cloud data into a plane coordinate system where the plane image is located to obtain a projected image;
[0118] A matching module 630, configured to modify the projection image based on the similarity between the projection image and the plane image to obtain an optimized projection image;
[0119] A rigid body transformation matrix optimization module 640, configured to obtain an optimized rigid body transformation matrix based on a mapping relationship between the point cloud data and the optimized projection image;
[0120] The registration module 650 is used to automatically register the point cloud data and the plane image based on the optimized rigid transformation matrix, and output a registration result.
[0121] Furthermore, the matching module 630 includes a similarity calculation unit 631 and a matching optimization unit 632, wherein the similarity calculation unit 631 is used to construct a cost function based on the similarity between the projection image and the plane image; the matching optimization unit 632 is used to iteratively correct the projection image until the similarity of the cost function meets a preset similarity threshold, thereby obtaining an optimized projection image.
[0122] Specifically, in this embodiment, the specific functions of the above-mentioned automatic radar-visual space registration system based on image feature learning can also refer to the corresponding description in the above-mentioned automatic radar-visual space registration method based on image feature learning, and will not be repeated here.
[0123] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 7As shown. The terminal includes a processor, a memory, a network interface and a display screen connected via a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a radar-visual space automatic registration program based on image feature learning. The internal memory provides an environment for the operation of the operating system and the radar-visual space automatic registration program based on image feature learning in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the radar-visual space automatic registration program based on image feature learning is executed by the processor, the steps of any one of the above-mentioned radar-visual space automatic registration methods based on image feature learning are implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.
[0124] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0125] In one embodiment, a terminal is provided, comprising a memory, a processor, and a radar-visual space automatic registration program based on image feature learning stored in the memory and executable on the processor. When the radar-visual space automatic registration program based on image feature learning is executed by the processor, the steps of any one of the radar-visual space automatic registration methods based on image feature learning provided in an embodiment of the present invention are implemented.
[0126] An embodiment of the present invention further provides a computer-readable storage medium, on which is stored a program for automatic radar-visual space registration based on image feature learning. When the program for automatic radar-visual space registration based on image feature learning is executed by a processor, the steps of any one of the methods for automatic radar-visual space registration based on image feature learning provided in an embodiment of the present invention are implemented.
[0127] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0128] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0131] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0132] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A method for automatic registration of radar-visual space based on image feature learning, characterized in that: Scenarios for target detection and tracking in intelligent transportation include: Collect the point cloud data of the millimeter wave radar and the plane image of the visual camera at the same time; Initialize a rigid body transformation matrix between the radar and the visual camera, and based on the rigid body transformation matrix, project the point cloud data into a plane coordinate system where the plane image is located to obtain a projected image; Based on the similarity between the projection image and the plane image, the projection image is corrected to obtain an optimized projection image; Based on the mapping relationship between the point cloud data and the optimized projection image, an optimized rigid body transformation matrix is obtained; Based on the optimized rigid body transformation matrix, automatically aligning the point cloud data and the plane image, and outputting the alignment result; The step of projecting the point cloud data onto a plane coordinate system where the plane image is located based on the rigid body transformation matrix to obtain a projected image includes: Obtain the straight-line distance from the target object to the plane coordinate system, and obtain the depth of the projected pixel; obtain the intrinsic parameter matrix of the visual camera, and establish a mapping relationship between a single pixel of the point cloud data and the plane coordinate system based on the intrinsic parameter matrix, the depth of the projected pixel and the rigid body transformation matrix to obtain a projection equation; based on the projection equation, project the point cloud data into the plane coordinate system to obtain a projection image; The step of correcting the projection image based on the similarity between the projection image and the plane image to obtain an optimized projection image includes: Based on the similarity between the projection image and the plane image, a cost function is constructed; based on the cost function, the projection image is corrected to obtain an optimized similarity; and the step of correcting the projection image is repeated until the optimized similarity meets a preset similarity threshold, thereby obtaining an optimized projection image; The constructing a cost function based on the similarity between the projection image and the plane image comprises: Feature extraction is performed on the projection image and the plane image respectively to obtain a projection feature map and a plane feature map; the projection feature map and the plane feature map are flattened to obtain a projection feature vector and a plane feature vector respectively; and a cost function is constructed based on the similarity between the position vectors of each pixel of the projection feature vector and the plane feature vector within a preset pixel range.
2. The method for automatic radar-visual space registration based on image feature learning according to claim 1 is characterized in that: The step of correcting the projected image based on the cost function to obtain an optimized similarity includes: Correcting the projected image within a preset pixel range to obtain a corrected projected image; The similarity between the corrected projection image and the plane image represented by the cost function within the pixel range is solved to obtain an optimized similarity.
3. The method for automatic radar-visual space registration based on image feature learning according to claim 1 is characterized in that: The step of obtaining an optimized rigid body transformation matrix based on a mapping relationship between the point cloud data and the optimized projection image comprises: Based on the projection equation and the depth of the projection pixel, mapping the optimized projection image to a three-dimensional space to obtain reconstructed point cloud data; A rigid body transformation is performed based on the reconstructed point cloud data and the projection image to obtain an optimized rigid body transformation matrix.
4. The method for automatic radar-visual space registration based on image feature learning according to claim 1 is characterized in that: After outputting the registration result, the registration error is calculated, which specifically includes: Based on the optimized rigid body transformation matrix, obtaining an extrinsic parameter prediction value and an extrinsic parameter true value; Acquire the initial extrinsic parameters of the radar, and construct a point cloud distance loss function based on the point cloud data, the initial extrinsic parameters, the extrinsic parameter prediction value and the extrinsic parameter true value; Based on the point cloud distance loss function, a registration error is obtained.
5. A radar-visual space automatic registration system based on image feature learning, characterized in that: The method for realizing the automatic registration of the radar-visual space based on image feature learning as claimed in claims 1 to 4 comprises: The information acquisition module is used to collect the point cloud data of the radar and the plane image of the visual camera at the same time; A projection module, used for initializing a rigid body transformation matrix between the radar and the visual camera, and projecting the point cloud data into a plane coordinate system where the plane image is located based on the rigid body transformation matrix to obtain a projected image; A matching module, used for correcting the projection image based on the similarity between the projection image and the plane image to obtain an optimized projection image; A rigid body transformation matrix optimization module, used for obtaining an optimized rigid body transformation matrix based on a mapping relationship between the point cloud data and the optimized projection image; A registration module is used to automatically register the point cloud data and the plane image based on the optimized rigid body transformation matrix, and output a registration result.
6. A terminal, characterized in that: The terminal includes a memory, a processor, and a radar-visual space automatic registration program based on image feature learning stored in the memory and executable on the processor. When the radar-visual space automatic registration program based on image feature learning is executed by the processor, the steps of the radar-visual space automatic registration method based on image feature learning as described in any one of claims 1 to 4 are implemented.
Citation Information
Patent Citations
3D / 2D medical image registration method based on LoG operator
CN111080681A
360-degree three-dimensional reconstruction optimization method based on continuous phase dense matching
CN111242990A
Single-lens three-dimensional image reconstruction method based on laser radar point cloud data assistance
CN112102458A
Method and system for registering laser point cloud and image
CN112767458A
Phased array laser radar imaging method based on RANSAC (Random Sample Consensus) and ICP (Inductively Coupled Plasma) point cloud registration
CN115205354A
