Vehicle exterior image registration method, device and electronic equipment

Abstract

This invention provides a method, apparatus, and electronic device for vehicle exterior image registration. The method includes: determining a target region from a 3D point cloud corresponding to a template image; determining a region to be matched from a 3D point cloud corresponding to an image to be detected; obtaining a first local feature descriptor corresponding to points in the target region and a second local feature descriptor corresponding to points in the region to be matched; obtaining a first transformation matrix based on the first and second local feature descriptors; obtaining a third transformation matrix based on the first transformation matrix; performing a two-dimensional perspective transformation on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected; and transforming the 2D image to be detected using the pseudo-3D transformation matrix to obtain the registered image to be detected. This invention can accurately register 2D images to be detected, improving the accuracy of vehicle exterior image detection.

Description

Technical Field

[0001] This invention relates to the field of rail transit equipment manufacturing technology, and provides a vehicle appearance image registration method, apparatus, and electronic device. Background Technology

[0002] In the field of rail transit equipment manufacturing, the core purpose of train inspection robots repeatedly inspecting the same component is to achieve a leap from static detection to predictive maintenance through data accumulation and dynamic comparison. When train inspection robots repeatedly inspect the same component of a train, due to navigation issues and deviations in parking positions, the robot may experience pose shifts during image acquisition. For example, it may shift along the vehicle direction or rotate along the X, Y, and Z axes. This pose difference leads to a discrepancy between the acquired image and the template image. Detection algorithms have high detection accuracy with template images. For example, detection algorithms are used to perform tasks such as detecting damage to critical components or predicting lifespan. If the deviation between the real-time acquired image to be inspected and the template image is large, it may cause subsequent detection algorithms to misjudge or fail to identify the position of critical components, affecting the detection accuracy of various detection tasks. Summary of the Invention

[0003] This invention provides a vehicle exterior image registration method, apparatus, and electronic device to solve the problem of deviation between the image to be detected and the template image affecting detection accuracy.

[0004] This invention proposes a vehicle exterior image registration method, comprising: determining a target region from a 3D point cloud corresponding to a template image; determining a region to be matched from a 3D point cloud corresponding to an image to be detected; obtaining a first local feature descriptor corresponding to a point in the target region and a second local feature descriptor corresponding to a point in the region to be matched; obtaining a first transformation matrix based on the first and second local feature descriptors; obtaining a third transformation matrix based on the first transformation matrix; performing a two-dimensional perspective transformation on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected; and transforming the 2D image to be detected using the pseudo-3D transformation matrix to obtain the registered image to be detected.

[0005] According to one embodiment of the present invention, after determining the target region from the 3D point cloud corresponding to the template image and determining the region to be matched from the 3D point cloud corresponding to the image to be detected, and before obtaining the first local feature descriptor corresponding to the points in the target region and the second local feature descriptor corresponding to the points in the region to be matched, the method further includes: sampling the point cloud corresponding to the target region and the point cloud corresponding to the region to be matched based on a preset sampling distance and sampling ratio to obtain a plurality of first key points corresponding to the target region and a plurality of second key points corresponding to the region to be matched.

[0006] According to one embodiment of the present invention, obtaining a first local feature descriptor corresponding to a point in a target region and a second local feature descriptor corresponding to a point in a region to be matched includes: obtaining the first local feature descriptor corresponding to a point in a target region and the second local feature descriptor corresponding to a point in a region to be matched using the three-dimensional local feature descriptor SHOT.

[0007] According to an embodiment of the present invention, obtaining a first local feature descriptor corresponding to a point in a target region using the 3D local feature descriptor SHOT includes: establishing a first local reference coordinate system centered on each of a plurality of first key points included in the target region; dividing a 3D space of a preset size around the first key point in the first local reference coordinate system to obtain a plurality of first 3D units corresponding to a first key point; generating a first local histogram corresponding to each of the plurality of first 3D units; and stitching together and normalizing the plurality of first local histograms corresponding to the plurality of first 3D units corresponding to a first key point to obtain a first local feature descriptor corresponding to a first key point. The method involves obtaining a second local feature descriptor for a point in the region to be matched using the 3D local feature descriptor SHOT, including: establishing a second local reference coordinate system centered on each of the multiple second key points in the region to be matched; dividing the 3D space of a preset size around the second key point in the second local reference coordinate system to obtain multiple second 3D units corresponding to one second key point; generating a second local histogram corresponding to each of the multiple second 3D units; and concatenating and normalizing the multiple second local histograms corresponding to the multiple second 3D units corresponding to one second key point to obtain a second local feature descriptor corresponding to one second key point.

[0008] According to one embodiment of the present invention, obtaining a first transformation matrix based on a first local feature descriptor and a second local feature descriptor includes: determining a one-to-one correspondence between a plurality of first key points included in the target region and a plurality of second key points included in the region to be matched based on the similarity between the first local feature descriptor and the second local feature descriptor; and obtaining the first transformation matrix based on the one-to-one correspondence using the Random Sample Consensus (RANSAC) method.

[0009] According to an embodiment of the present invention, obtaining a third transformation matrix based on a first transformation matrix includes: performing a preliminary pose transformation on a plurality of first key points included in the target region using the first transformation matrix to obtain an initial template point cloud; using the iterative nearest point (ICP) method to find the minimum distance between the initial template point cloud and a plurality of second key points included in the region to be matched to obtain a second transformation matrix; and merging the first transformation matrix and the second transformation matrix to obtain a third transformation matrix.

[0010] According to one embodiment of the present invention, determining a target region from a 3D point cloud corresponding to a template image includes: selecting a region with significant appearance features from the 3D point cloud corresponding to the template image as the target region; the region with significant appearance features includes a high curvature region and / or a region with local geometric structure; the high curvature region includes points with curvature exceeding a preset value; the local geometric structure includes surfaces and / or corner points with regular or irregular geometric shapes.

[0011] This invention also provides a vehicle exterior image registration device, comprising: a 3D point cloud processing module, used to determine a target region from the 3D point cloud corresponding to a template image; and to determine a region to be matched from the 3D point cloud corresponding to an image to be detected; a descriptor module, used to obtain a first local feature descriptor corresponding to a point in the target region and a second local feature descriptor corresponding to a point in the region to be matched; an initial transformation module, used to obtain a first transformation matrix based on the first and second local feature descriptors; a precise transformation module, used to obtain a third transformation matrix based on the first transformation matrix; a perspective transformation module, used to perform a two-dimensional perspective transformation on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected; and a registration module, used to transform the 2D image to be detected using the pseudo-3D transformation matrix to obtain a registered image to be detected.

[0012] The present invention also provides an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the vehicle appearance image registration method as described above.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a vehicle appearance image registration method as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a vehicle appearance image registration method as described above.

[0015] This invention provides a vehicle exterior image registration method, apparatus, and electronic device. The method determines a target region from the 3D point cloud corresponding to a template image and a region to be matched from the 3D point cloud corresponding to the image to be detected. Then, a first local feature descriptor and a second local feature descriptor are obtained. Based on the first and second local feature descriptors, a first transformation matrix is ​​obtained, which is a preliminary transformation matrix. A third transformation matrix is ​​further obtained based on the first transformation matrix, and a two-dimensional perspective transformation is performed on the third transformation matrix to obtain a pseudo-3D transformation matrix. The pseudo-3D transformation matrix is ​​then used to transform the 2D image to be detected to obtain the registered image to be detected. This method utilizes 3D point cloud data of the vehicle exterior for image registration. The local feature descriptors obtained based on the 3D point cloud data contain richer and more three-dimensional local feature information of the vehicle exterior. Compared with the planar 2D image, it has more depth information, thus making the transformation matrix obtained based on the local feature descriptors more closely match the actual three-dimensional shape of the vehicle exterior, thereby improving the accuracy of image registration and obtaining a more standard image to be detected. Inputting the registered image to be detected into the detection algorithm can improve the detection accuracy of various detection tasks such as surface defect detection, connection status monitoring, and structural integrity detection of key components. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the vehicle exterior image registration method provided by the present invention.

[0018] Figure 2 This is a flowchart illustrating an embodiment of the vehicle exterior image registration method provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the process for obtaining the first local feature descriptor in one embodiment of the vehicle appearance image registration method provided by the present invention.

[0020] Figure 4 This is a schematic diagram of the process for obtaining the second local feature descriptor in one embodiment of the vehicle appearance image registration method provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the software module structure of the vehicle exterior image registration device provided by the present invention.

[0022] Figure 6This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0023] Figure label: 501. 3D Point Cloud Processing Module; 502. Description Submodule; 503. Initial Transformation Module; 504. Precise Transformation Module; 505. Perspective Transformation Module; 506. Registration Module; 610. Processor; 620. Communication interface; 630. Memory; 640. Communication bus. Detailed Implementation

[0024] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0025] Train inspection robots are intelligent equipment integrating mobile robot technology, multi-sensor fusion, artificial intelligence, and the Internet of Things (IoT). They are specifically designed for the daily inspection, condition monitoring, and predictive maintenance of rail transit vehicles. The train inspection robot inspects various components of the train, collecting images of key components as inspection images. Detection algorithms then perform inspections on these images. The core task of these algorithms is to automatically and quantitatively identify and measure anomalies in the visual state of key components. For example, they detect whether key components have surface defects (cracks, peeling, corrosion, dents, etc.), structural integrity (missing components, deformed / bent components, foreign matter attachment, etc.), and connection status (loose bolts, welding defects, aged / damaged seals, etc.).

[0026] When a train inspection robot repeatedly inspects the same component of a train, due to navigation issues and inconsistent stopping conditions, the image may shift along the vehicle's direction and rotate along the X, Y, and Z axes during the acquisition process. These differences can cause the subsequent detection algorithm to deviate in judging the position of key components, affecting the detection accuracy of the task. Therefore, it is necessary to register the currently acquired images to be inspected to keep the positions of key components consistent in order to ensure the accuracy of subsequent detection.

[0027] In view of this, embodiments of the present invention propose a vehicle exterior image registration method, apparatus, and electronic device for registering images to be detected, so as to improve detection accuracy.

[0028] The vehicle exterior image registration method, apparatus, and electronic device proposed in the embodiments of the present invention will be described below with reference to the accompanying drawings. The vehicle exterior image registration method proposed in the embodiments of the present invention, such as... Figure 1 As shown, the method includes: Step 101: Determine the target region from the 3D point cloud corresponding to the template image; determine the region to be matched from the 3D point cloud corresponding to the image to be detected.

[0029] The 3D vision sensor mounted on the inspection robot can collect 3D (Three-Dimensional) point cloud data of the train's exterior. The template image refers to a standard reference image, which can be understood as a standard image of a key train component acquired under ideal conditions (precise robot positioning and correct angle). Subsequent newly captured images must be aligned with the template image. The template image can come from historical data or initial calibration and contains the correct position and shape of the component. The corresponding 3D point cloud, i.e., the template in 3D point cloud format, serves as the standard reference point cloud.

[0030] The image to be detected is a new image acquired in real time during the inspection process. It is an image that needs to be processed, registered, and ultimately used as input to the detection algorithm to perform the detection task. The 3D point cloud corresponding to the image to be detected is a new 3D point cloud that needs to be registered and acquired in real time during the inspection process.

[0031] The target region determined from the 3D point cloud corresponding to the template image is a three-dimensional spatial region. Specifically, it can be a region with significant appearance features selected from the 3D point cloud corresponding to the template image as the target region. In other words, the target region is a region in the 3D point cloud with significant appearance features. A region with significant appearance features refers to a point cloud region in three-dimensional space that has a unique and stable geometry, is not easily affected by slight changes in viewing angle, and is easily distinguishable from other regions.

[0032] For example, regions with significant appearance features can include high-curvature regions and / or regions with local geometric structures. High-curvature regions include points with curvature exceeding a preset value (points in a point cloud), such as corners, edges, protrusions, or depressions; these regions have very rich geometric information. Local geometric structures include faces and / or corners of regular or irregular geometric shapes, such as a small plane, a regular cylindrical surface (like a bolt head), or the corner of a cube—regions whose basic shape characteristics remain unchanged even from different angles.

[0033] It should be noted that areas with significant appearance features should be identified as areas with low repeatability. In other words, the appearance features of areas with significant appearance features are unique and it is difficult to find similar shapes in other areas of the template drawing. This can avoid misidentifying objects during registration.

[0034] The matching region is determined from the 3D point cloud corresponding to the image to be detected. Specifically, this can be a larger region obtained by expanding upon the target region. In other words, a wider matching region is selected from the 3D point cloud to be detected and used to match the target region in the template image. Since the image to be detected has offset and rotation, it is impossible to know in advance the exact offset position of the target region corresponding to the template image. Therefore, it is necessary to search and match within a larger range that may contain the target region. This larger range is the matching region.

[0035] Step 102: Obtain the first local feature descriptor corresponding to the point in the target region and the second local feature descriptor corresponding to the point in the region to be matched.

[0036] In this embodiment of the invention, the local feature can be a point feature, and the local feature descriptor can also be called a point feature descriptor or a point descriptor.

[0037] A local feature descriptor is a digital and unique vector or descriptor that describes the local appearance features of a keypoint extracted from a 3D point cloud, within a pre-defined spatial range surrounding that point. The core objective of local feature descriptors is to ensure that the descriptors calculated for the same physical point are as similar as possible under different viewpoints, lighting conditions, and scales in an image, while making the descriptors for different physical points as different as possible.

[0038] The first local feature descriptor is the local feature descriptor corresponding to each keypoint in the 3D point cloud of the target region. The second local feature descriptor is the local feature descriptor corresponding to each keypoint in the 3D point cloud of the region to be matched.

[0039] Specifically, descriptors can be constructed by statistically analyzing the orientation histogram of pixel gradients within the neighborhood of keypoints. For example, keypoints can be determined from the 3D point clouds corresponding to the target region or the region to be matched. Keypoint determination can be achieved through sampling or keypoint detection. For instance, keypoints can be obtained by sampling the 3D point cloud data using a preset sampling method. For example, a specific sampling method could be one of the following: Voxelized mesh sampling: Divide the three-dimensional space into a uniform, tiny cubic mesh (voxels), and then use the center point or centroid of all points within each voxel to represent that voxel; Random sampling: Selecting a certain proportion or number of points from the original point cloud as key points by purely random sampling; Uniform sampling: Key points in the point cloud are selected using a uniform sampling method, and the sampling distance D and sampling ratio f are set.

[0040] Key point detection can further detect corner points, edge intersections, and points with rich textures from the target area as key points, and filter out other points.

[0041] The vehicle exterior images collected by the inspection robot have high resolution and large point cloud data volume. If the local feature descriptor is calculated directly on the original 3D point cloud data, the computational load is large. Calculating the local feature descriptor of the key points after extracting them can reduce the consumption of computing resources and reduce the computational load.

[0042] After determining the key points, feature description is performed: around each key point, the volumetric pixel information (such as gradient, intensity, color, etc.) of its local neighborhood (such as a spherical or cubic spatial region) is analyzed and encoded into a fixed-length numerical vector, which is the descriptor.

[0043] For example, one or more of SHOT (Signature of Histograms of OrienTations), SIFT (Scale-Invariant Feature Transform), or SURF (Speeded-Up Robust Features, an accelerated version of SIFT) can be used to obtain the first local feature descriptor and the second local feature descriptor.

[0044] For example, in some embodiments, the three-dimensional local feature descriptor SHOT can be used to obtain a first local feature descriptor corresponding to a point in the target region and a second local feature descriptor corresponding to a point in the region to be matched.

[0045] Step 103: Obtain the first transformation matrix based on the first local feature descriptor and the second local feature descriptor.

[0046] In some embodiments, a first transformation matrix is ​​obtained based on a first local feature descriptor and a second local feature descriptor. Specifically, this can be achieved by determining a one-to-one correspondence between multiple first keypoints in the target region and multiple second keypoints in the region to be matched, based on the similarity between the first and second local feature descriptors. For example, two keypoints with the highest or highest similarity (the first keypoint and the second keypoint) are identified as having a one-to-one correspondence. Then, based on this one-to-one correspondence, RANSAC (RANdom Sampling Consensus) is applied to obtain the first transformation matrix. The first transformation matrix is ​​used to represent the spatial transformation information of the movement from the second keypoint to the first keypoint position between the two types of keypoints with a corresponding relationship.

[0047] Step 104: Based on the first transformation matrix, obtain the third transformation matrix.

[0048] The first transformation matrix is ​​a preliminary transformation matrix used to achieve a coarse match between the target region and the region to be matched. In some embodiments, based on the first transformation matrix, a second transformation matrix is ​​further obtained. The second transformation matrix is ​​a precise transformation matrix used to achieve a more precise match based on the coarse match. Then, based on the first transformation matrix, a third transformation matrix is ​​obtained. In some embodiments, the first and second transformation matrices can be combined to obtain the third transformation matrix. In other embodiments, the first transformation matrix can be directly used as the third transformation matrix, i.e., precise matching is not performed.

[0049] Step 105: Perform a two-dimensional perspective transformation on the third transformation matrix to obtain the pseudo-3D transformation matrix corresponding to the 2D image to be detected.

[0050] The third transformation matrix contains the spatial position transformation information for registering the 3D point cloud of the region to be matched corresponding to the image to be detected to the 3D point cloud of the target region corresponding to the template image. However, most current detection algorithms support 2D (Two-Dimensional) image input and detection. To ensure compatibility with detection algorithms, the third transformation matrix needs to be transformed into planar position transformation information between 2D images. Specifically, the registration process is a rigid pose transformation. The projection of the 2D image into a 3D transformation is used to perform a 2D perspective transformation on the third transformation matrix, resulting in a pseudo-3D transformation matrix corresponding to the 2D image.

[0051] Step 106: Transform the 2D image to be detected using a pseudo-3D transformation matrix to obtain the registered image to be detected.

[0052] The pseudo-3D transformation matrix is ​​the planar position transformation information between two-dimensional images. By using the pseudo-3D transformation matrix to transform the 2D image to be detected, the registered image to be detected can be obtained. In this way, the registration process of the image to be detected is completed.

[0053] Figure 1The illustrated embodiment utilizes 3D point cloud data of the vehicle exterior for image registration. 3D matching is performed between the region to be matched in the image to be detected and the target region in the template image to obtain a third transformation matrix. This third transformation matrix is ​​then converted into a projection transformation (pseudo-3D transformation matrix) of the synchronously generated 2D image. Registration between 2D images is then performed based on the pseudo-3D transformation matrix. This process utilizes 3D point cloud data of the vehicle exterior for image registration. The local feature descriptors obtained from the 3D point cloud data contain richer and more three-dimensional local feature information about the vehicle exterior, providing depth information compared to planar 2D images. This makes the transformation matrices obtained based on the local feature descriptors (the first transformation matrix and other transformation matrices obtained based on the first transformation matrix) more accurate, thereby improving the registration accuracy. Inputting the registered image to be detected into the detection algorithm can improve the detection accuracy of various detection tasks, such as surface defect detection of key components, connection status monitoring, and structural integrity detection.

[0054] The following are some specific examples.

[0055] like Figure 2 As shown, in one specific embodiment, the vehicle exterior image registration method may include the following process: Step 201: Determine the target region from the 3D point cloud corresponding to the template image; determine the region to be matched from the 3D point cloud corresponding to the image to be detected.

[0056] The target region and the region to be matched are determined from the 3D point cloud data corresponding to the image to be inspected and the template image, respectively. The target region selected from the 3D point cloud data corresponding to the template image (hereinafter referred to as the template point cloud) is a point cloud with significant appearance features, such as the 3D point cloud data corresponding to the bogie axle box area. The region to be matched in the image to be inspected is a wider range, used to find the region that can be registered with the target region.

[0057] Step 202: Based on the preset sampling distance and sampling ratio, sample the point cloud corresponding to the target area and the point cloud corresponding to the area to be matched to obtain multiple first key points corresponding to the target area and multiple second key points corresponding to the area to be matched.

[0058] Because the acquired vehicle exterior images have high resolution and large point cloud data volume, sampling processing is required. This embodiment uses a uniform sampling method to select key points in the point cloud, for example, by uniformly sampling according to a set sampling distance D and sampling ratio f. The sampling distance D represents the spatial distance between the current sampling point and its adjacent sampling points, and the sampling ratio represents the ratio of the number of key points obtained after sampling to the number of points in the original point cloud data before sampling. For example, a sampling ratio of 1 / 10 means that one point is sampled from 10 points in the point cloud as a key point.

[0059] In other words, in this embodiment, the sampled points are directly used as key points. For example, multiple first key points are obtained by sampling the original 3D point cloud data of the target region, and multiple second key points are obtained by sampling the original 3D point cloud data of the region to be matched. In other embodiments, key point detection is further performed based on the sampling, that is, key points are further detected from the multiple sampled points, and the detected points are used as first or second key points. Key point detection can be implemented based on a 3D point cloud key point detection algorithm.

[0060] Step 203: Obtain the first local feature descriptor corresponding to the point in the target region and the second local feature descriptor corresponding to the point in the region to be matched.

[0061] In this embodiment, the SHOT method is used to calculate the local feature descriptors of each key point in the template image and the detection image. The main process includes establishing a local reference coordinate system, performing spatial division, calculating the local histogram, and splicing and normalizing the descriptors.

[0062] Specifically, such as Figure 3 As shown, the first local feature descriptor corresponding to a point in the target region can be obtained using the 3D local feature descriptor SHOT, which can be implemented in the following way: Step 301: Establish a first local reference coordinate system with each of the multiple first key points included in the target area as the center.

[0063] For example, with a first key point as the center, a unique and stable three-dimensional coordinate system (X-axis, Y-axis, Z-axis) is calculated based on the positions of the adjacent first key points around it. This coordinate system will change with the surface normal and curvature direction, but for the same geometry, the direction of the calculated coordinate system is consistent from any angle, so that the description has rotation invariance.

[0064] Step 302: In the first local reference coordinate system, the 3D space around the first key point of a preset size is divided into multiple first 3D units corresponding to the first key point.

[0065] For example, in the newly established local coordinate system, a spherical space with radius R is drawn, centered on the first key point. This sphere is then uniformly divided into multiple small grids along the radial, longitude, and latitude directions. Each small grid is a 3D unit (or voxel). For example, if the radial direction is divided into 2 layers, the longitude into 8 parts, and the latitude into 2 parts, there are a total of 2 × 8 × 2 = 32 3D units. After division, the space can be encoded to record the distribution of geometric information within the space.

[0066] Step 303: Generate the first local histogram corresponding to each of the multiple first 3D units.

[0067] For example, for each small grid (3D unit) divided in step 302, the normal directions of all first keypoints falling within this grid are statistically analyzed. Specifically, the cosine of the angle between the normal vector of each first keypoint and the Z-axis of the first local reference coordinate system (usually the surface normal) is calculated. Then, this cosine value range (-1 to 1) is divided into several intervals (e.g., 11 intervals) to form a statistical histogram. This histogram represents the distribution of surface directions within the small grid. A histogram is a quantitative description of geometry. In a flat region, the normals of all points are almost identical, and the histogram will concentrate on a single peak; in a curved and complex region, the normals vary greatly, and the histogram will be more dispersed.

[0068] Step 304: Concatenate and normalize the multiple first local histograms corresponding to multiple first 3D units corresponding to a first key point to obtain a first local feature descriptor corresponding to a first key point.

[0069] Next, the histograms calculated from the multiple small grid cells within the spherical space of radius R are concatenated end-to-end according to their spatial position, forming a very long, one-dimensional feature vector. This vector is the SHOT descriptor. Finally, this long vector is normalized, for example, by performing L2 normalization, which sets the magnitude of the entire vector to 1. The normalized vector is the first local feature descriptor. L2 normalization, also known as Euclidean normalization or unit vectorization, is essentially scaling a non-zero vector so that its L2 norm (magnitude) becomes 1, while maintaining its direction.

[0070] Normalization is used to ensure invariance to illumination and scale (changes in point cloud density). Normalization allows the descriptor to focus only on the "pattern" of shape, without being affected by the absolute number of points in the cloud.

[0071] Similarly, such as Figure 4 As shown, the second local feature descriptor corresponding to a point in the region to be matched can be obtained using the 3D local feature descriptor SHOT, which can be implemented in the following way: Step 401: Establish a second local reference coordinate system centered on each of the multiple second key points included in the region to be matched.

[0072] Step 402: In the second local reference coordinate system, the 3D space around the second key point of a preset size is divided into multiple second 3D units corresponding to a second key point.

[0073] Step 403: Generate the second local histogram corresponding to each of the multiple second 3D units.

[0074] Step 404: Concatenate and normalize the multiple second local histograms corresponding to multiple second 3D units corresponding to a second keypoint to obtain a second local feature descriptor corresponding to a second keypoint.

[0075] The specific implementation methods of steps 401-404 can be referred to steps 301-304 above, and will not be repeated here.

[0076] In this way, each key point corresponding to the template image and the image to be detected is transformed into a fixed-length, high-dimensional digital vector (local feature descriptor).

[0077] Step 204: Obtain the first transformation matrix based on the first local feature descriptor and the second local feature descriptor.

[0078] In this embodiment, based on the first local feature descriptor and the second local feature descriptor obtained in the above steps, the correspondence between the second key point corresponding to the image to be detected and the first key point corresponding to the template image is calculated. This process is the feature matching process.

[0079] If two keypoints are the same physical point (or highly similar structural points) in 3D space, then the vectors of their SHOT descriptors (first local feature descriptor and second local feature descriptor) will be mathematically very close. Using this principle, the correspondence between the first and second keypoints can be determined based on the local feature descriptors obtained in the above steps. For example, the specific implementation is as follows: First, calculate the distance / similarity between a first local feature descriptor and a second local feature descriptor: for example, calculate the Euclidean distance or cosine similarity between them. Since the first or second local feature descriptor has already been L2 normalized, the Euclidean distance or cosine similarity between the two descriptor vectors can be calculated efficiently. The smaller the Euclidean distance (or the larger the cosine similarity), the more similar the local geometry represented by the two descriptors. Specifically, for a key point A in the target region (the target region corresponding to the template image), its first local feature descriptor (hereinafter referred to as descriptor) is Desc_A. Traverse the second local feature descriptors (hereinafter referred to as descriptors) Desc_B1, Desc_B2, ... Desc_Bn of all key point detection images included in the region to be matched (the region to be matched corresponding to the detection image), and calculate the distance between Desc_B1, Desc_B2, ... Desc_Bn and Desc_A respectively. Find the second key point with the smallest distance, such as B_k. Then determine that there is a one-to-one correspondence between the first key point A and the second key point B_k.

[0080] Optionally, to improve matching accuracy, mismatched point pairs can be removed. In practical applications, there may be multiple different parts with similar local shapes (such as two different bolt heads). To increase the reliability of feature matching, a nearest neighbor distance ratio strategy is used to remove mismatches: Find the keypoint B_first (distance dist_first) with the smallest distance to Desc_A and the second smallest keypoint B_second (distance dist_second), and calculate the ratio ratio = dist_first / dist_second. Set a threshold (such as 0.6 or 0.8). If the ratio is less than this threshold, the (A, B_first) pair is considered a valid match, meaning there is a one-to-one correspondence between the first keypoint A and the second keypoint B_first. If the ratio is too large, exceeding the threshold (e.g., 0.6 or 0.8), the match is considered unreliable and is removed.

[0081] After obtaining the one-to-one correspondence between the first and second key points, the Random Sampling Consensus (RANSAC) method is used to obtain the preliminary transformation matrix H1 (the first transformation matrix).

[0082] Specifically, RANSAC is employed, using a random sampling and labeling strategy to resist the interference of erroneous data (mismatched point pairs), thereby estimating a robust transformation matrix. Although some mismatched point pairs are eliminated to improve matching accuracy, some mismatched point pairs may still exist. Alternatively, in some embodiments, the above-mentioned operation of eliminating mismatched point pairs may not be performed, and the RANSAC algorithm may be used to resist the interference of erroneous data instead.

[0083] Specifically, assuming that N first keypoints in the target region match N second keypoints in the region to be matched one-to-one, there are N pairs of matching points, including both correct and incorrect matches. Specifically, the following iterative process is executed: (1) Random sampling: Randomly and uniformly extract the minimum sample set from all N pairs of matching points. For three-dimensional rigid body transformation (rotation + translation), for example, the minimum number of samples is 3 pairs of non-collinear key points (3 pairs of non-collinear 3D points can uniquely determine a rigid body transformation).

[0084] (2) Model calculation: Using 3 pairs of non-collinear key points, a rigid body transformation matrix H_temp is calculated by mathematical methods (such as SVD, Singular Value Decomposition). This matrix can align the above 3 pairs of non-collinear key points (with zero distance).

[0085] (3) Model validation (voting): Use this temporary matrix H_temp to test all N pairs of matching points. For each pair, transform the first key point corresponding to the template image with H_temp, calculate the Euclidean distance between the first key point and the second key point corresponding to the detection image. If the Euclidean distance is less than the preset threshold, increment the voting value of the temporary matrix H_temp by 1, and mark the corresponding matching point pair as an interior point.

[0086] (4) Obtain the first transformation matrix: record the transformation matrix H_temp that has the most interior points in this iteration and its corresponding set of interior points.

[0087] After the iteration is completed, the transformation matrix H_temp generated by the iteration with the most interior points is selected as the first transformation matrix H1.

[0088] Based on the above exemplary description, it can be seen that in this embodiment, the first transformation matrix is ​​obtained based on the first local feature descriptor and the second local feature descriptor. Specifically, it can be that the one-to-one correspondence between multiple first key points included in the target region and multiple second key points included in the region to be matched is determined based on the similarity between the first local feature descriptor and the second local feature descriptor. Then, based on the one-to-one correspondence, the Random Sample Consensus (RANSAC) method is used to obtain the first transformation matrix.

[0089] Step 205: Based on the first transformation matrix, obtain the second transformation matrix, and merge the first transformation matrix and the second transformation matrix to obtain the third transformation matrix.

[0090] Based on the preliminary transformation matrix H1 obtained in step 204, a preliminary pose transformation is performed on the template point cloud (the 3D point cloud corresponding to the target region). That is, the first transformation matrix H1 is used to perform a preliminary pose transformation on multiple first keypoints included in the target region to obtain the initial template point cloud. Then, the ICP (Iterative Closest Point) method is used to find the minimum distance between the initial template point cloud and multiple second keypoints included in the region to be matched, to obtain the second transformation matrix. For example, the nearest neighbor calculation method of ICP is used to calculate the nearest distance between each first keypoint in the template point cloud and each second keypoint corresponding to the corresponding detection map. The process is iterated multiple times (e.g., 100-200 times) to find the pose transformation with the minimum sum of all point distances, to obtain the fine registration transformation matrix H2 (the second transformation matrix).

[0091] Specifically, after the initial pose transformation using the first transformation matrix in the above steps, a coarse match between the two point clouds is achieved. At this point, the two point clouds are very close, and then fine-tuning is performed using ICP to achieve optimal alignment. For example, assuming the template point cloud after the initial RANSAC alignment (after H1 transformation) is P, and the detection point cloud (i.e., the point cloud corresponding to the matching region in the image to be detected) is Q, the following iterative steps are used to achieve this: For each first keypoint in the transformed template point cloud P, find the closest second keypoint in the detection point cloud Q to form a new point pair (pi, qi). Next, calculate the sum of squared distances between all matching point pairs as the current alignment error: E = Σ ||pi - qi || 2Then, a transformation solution is performed, namely, solving for a new rigid body transformation matrix H_iter, such that the error E calculated in the previous iteration step is minimized after applying the transformation matrix H_iter. For example, the transformation matrix H_iter can be obtained through closed-form solutions such as SVD. Next, the solved transformation matrix H_iter is applied to the template point cloud P, resulting in the updated point cloud P' = H_iter * P. Then, it is checked whether the decrease in error E is less than a preset threshold (e.g., 0.001) or whether the maximum number of iterations (e.g., 200) has been reached. If it is not less than the preset threshold and the maximum number of iterations has not been reached, the next iteration begins with the updated P' and Q, until the convergence condition is met. The convergence condition is that the decrease in error E is less than the preset threshold or the maximum number of iterations has been reached.

[0092] After the iteration is completed, the transformation matrices applied in all the iteration steps are multiplied together to obtain the fine registration transformation matrix H2 from the initial input (i.e. the state after H1 transformation) to the final aligned state, that is: H2 = H_iter(k) ×H_iter(k-1) × ... ×H_iter(1), where k is the iteration number.

[0093] After obtaining the second transformation matrix H2, the final transformation matrix H3 is obtained by merging H1 and H2. Specifically, the first and second transformation matrices are merged to obtain the third transformation matrix, which can be obtained by multiplying the first and second transformation matrices.

[0094] The first transformation matrix H1 is used to transform the original, untransformed detection point cloud to a state that is initially aligned with the template point cloud; the second transformation matrix H2 is used to transform the detection point cloud that is already in the initial alignment state to a state that is finely aligned with the template point cloud. The total transformation required to transform directly from the original detection point cloud to the final finely aligned state is to first perform H1, then H2, that is, the third transformation matrix H3 = H2 × H1.

[0095] Step 206: Perform a two-dimensional perspective transformation on the third transformation matrix to obtain the pseudo-3D transformation matrix corresponding to the 2D image to be detected.

[0096] In this embodiment, the projection of a 2D image into a 3D transformation is used to perform a 2D perspective transformation on the pose transformation matrix (third transformation matrix) H3, resulting in a pseudo-3D transformation matrix T for the 2D image. For example, several (e.g., 4-6 or more) second keypoints are selected in the detection point cloud Q. Using the already calculated third transformation matrix H3, the coordinates of these second keypoints are transformed to the coordinates corresponding to the template image. The pixel coordinates of these two sets of keypoints (before and after transformation) projected onto their respective 2D images are calculated (using known camera intrinsic parameters and fixed extrinsic parameters). In this way, multiple pairs of 2D image pixel correspondences (p_template, p_detect) can be obtained. Next, using multiple pairs of 2D image pixel pairs, a system of linear equations (e.g., using Direct Linear Transformation, DLT) is solved to obtain a matrix T that makes p_template ≈ T * p_detect true. For example, matrix T is a matrix of minimum size 3×3. Thus, by utilizing the 3D spatial relative pose relationships contained in H3 and combining them with the camera model, the pure 2D perspective transformation matrix T (pseudo 3D transformation matrix) connecting two 2D images can be derived or calculated.

[0097] Step 207: Transform the 2D image to be detected using a pseudo-3D transformation matrix to obtain the registered image to be detected.

[0098] The method proposed in this invention first extracts local features (local feature descriptors) from 3D point clouds, performs 3D registration based on these local features to obtain 3D registration information (a third transformation matrix), and then performs pseudo-3D transformation on the synchronously generated 2D image to achieve the purpose of registering the image to be detected. Related technologies achieve fast registration of 2D images based on 2D image point features or template matching methods, but this results in low registration accuracy and poor registration effect, affecting detection accuracy. The vehicle exterior image registration method proposed in this invention extracts local feature descriptors from 3D point clouds and performs registration based on these local feature descriptors (e.g., point feature descriptors). Specifically, local feature descriptors are used to describe the local three-dimensional appearance features of key points corresponding to the template image and the image to be detected. Based on the local three-dimensional appearance features of the key points, RANSAC and ICP algorithms are combined to achieve feature matching and precise alignment between the template point cloud and the detection point cloud. Furthermore, two-dimensional perspective transformation is used to register the synchronously generated 2D image (simultaneously acquired by the inspection robot from 3D point clouds and 2D images), thereby improving the accuracy of subsequent detection algorithms in detecting key vehicle components. Thus, this method applies 3D data for registration to obtain a transformation matrix (third transformation matrix), and registers it with the synchronously generated 2D image to be detected. After actual testing, the method proposed in this embodiment of the invention has better image registration effect than the traditional method based on 2D point features and template matching. Inputting the registered 2D image into the detection algorithm can ensure that the detection accuracy of the detection algorithm is maintained at a high level, which has a significant accuracy improvement effect for subsequent vehicle appearance detection.

[0099] This invention also provides a vehicle exterior image registration device, which can be integrated into an inspection robot, for example, into the robot's intelligent agent. Figure 5 As shown, the device may include the following modules: The 3D point cloud processing module 501 is used to determine the target region from the 3D point cloud corresponding to the template image and to determine the region to be matched from the 3D point cloud corresponding to the image to be detected.

[0100] The descriptor submodule 502 is used to obtain the first local feature descriptor corresponding to the point in the target region and the second local feature descriptor corresponding to the point in the region to be matched.

[0101] The initial transformation module 503 is used to obtain the first transformation matrix based on the first local feature descriptor and the second local feature descriptor.

[0102] The precise transformation module 504 is used to obtain the third transformation matrix based on the first transformation matrix; The perspective transformation module 505 is used to perform two-dimensional perspective transformation on the third transformation matrix to obtain the pseudo-3D transformation matrix corresponding to the 2D image to be detected.

[0103] The registration module 506 is used to transform the 2D image to be detected using a pseudo-3D transformation matrix to obtain the registered image to be detected.

[0104] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus. The processor 610 can call logical instructions stored in the memory 630 to execute the following methods: The target region is determined from the 3D point cloud corresponding to the template image; the region to be matched is determined from the 3D point cloud corresponding to the image to be detected; the first local feature descriptor corresponding to the points in the target region and the second local feature descriptor corresponding to the points in the region to be matched are obtained; a first transformation matrix is ​​obtained based on the first and second local feature descriptors; a third transformation matrix is ​​obtained based on the first transformation matrix; a two-dimensional perspective transformation is performed on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected; the pseudo-3D transformation matrix is ​​used to transform the 2D image to be detected to obtain the registered image to be detected.

[0105] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] This invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as including: The target region is determined from the 3D point cloud corresponding to the template image; the region to be matched is determined from the 3D point cloud corresponding to the image to be detected; a first local feature descriptor corresponding to the points in the target region and a second local feature descriptor corresponding to the points in the region to be matched are obtained; a first transformation matrix is ​​obtained based on the first local feature descriptor and the second local feature descriptor; a third transformation matrix is ​​obtained based on the first transformation matrix; a two-dimensional perspective transformation is performed on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected; the pseudo-3D transformation matrix is ​​used to transform the 2D image to be detected to obtain the registered image to be detected.

[0107] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the transmission methods provided in the above embodiments, including, for example: The target region is determined from the 3D point cloud corresponding to the template image; the region to be matched is determined from the 3D point cloud corresponding to the image to be detected; a first local feature descriptor corresponding to the points in the target region and a second local feature descriptor corresponding to the points in the region to be matched are obtained; a first transformation matrix is ​​obtained based on the first local feature descriptor and the second local feature descriptor; a third transformation matrix is ​​obtained based on the first transformation matrix; a two-dimensional perspective transformation is performed on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected; the pseudo-3D transformation matrix is ​​used to transform the 2D image to be detected to obtain the registered image to be detected.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for registering vehicle exterior images, characterized in that, The method includes: The target region is determined from the 3D point cloud corresponding to the template image; the region to be matched is determined from the 3D point cloud corresponding to the image to be detected. Obtain the first local feature descriptor corresponding to the point in the target region and the second local feature descriptor corresponding to the point in the region to be matched; Based on the first local feature descriptor and the second local feature descriptor, the first transformation matrix is ​​obtained; Based on the first transformation matrix, the third transformation matrix is ​​obtained; Perform a two-dimensional perspective transformation on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected; The pseudo-3D transformation matrix is ​​used to transform the 2D image to be detected to obtain the registered image to be detected.

2. The method according to claim 1, characterized in that, The method further includes determining the target region from the 3D point cloud corresponding to the template image; and before obtaining the first local feature descriptor corresponding to the points in the target region and the second local feature descriptor corresponding to the points in the region to be matched, after determining the region to be matched from the 3D point cloud corresponding to the image to be detected, the method further includes: Based on a preset sampling distance and sampling ratio, the point cloud corresponding to the target region and the point cloud corresponding to the region to be matched are sampled to obtain multiple first key points corresponding to the target region and multiple second key points corresponding to the region to be matched.

3. The method according to claim 1 or 2, characterized in that, Obtaining the first local feature descriptor corresponding to a point in the target region and the second local feature descriptor corresponding to a point in the region to be matched includes: The three-dimensional local feature descriptor SHOT is used to obtain the first local feature descriptor corresponding to the point in the target region and the second local feature descriptor corresponding to the point in the region to be matched.

4. The method according to claim 3, characterized in that, The first local feature descriptor corresponding to a point in the target region is obtained using the 3D local feature descriptor SHOT, including: A first local reference coordinate system is established with each of the multiple first key points included in the target area as the center. In the first local reference coordinate system, the 3D space around the first key point of a preset size is divided into multiple first 3D units corresponding to a first key point. Generate a first local histogram corresponding to each of the plurality of first 3D units; By concatenating and normalizing the multiple first local histograms corresponding to multiple first 3D units corresponding to a first key point, a first local feature descriptor corresponding to a first key point is obtained. The second local feature descriptor corresponding to the point in the region to be matched is obtained by using the 3D local feature descriptor SHOT, including: A second local reference coordinate system is established with each of the multiple second key points included in the region to be matched as the center. In the second local reference coordinate system, the 3D space around the second key point of a preset size is divided into multiple second 3D units corresponding to a second key point. Generate a second local histogram corresponding to each of the plurality of second 3D units; By concatenating and normalizing the multiple second local histograms corresponding to multiple second 3D units corresponding to a second keypoint, a second local feature descriptor corresponding to a second keypoint is obtained.

5. The method according to claim 1, characterized in that, Based on the first local feature descriptor and the second local feature descriptor, a first transformation matrix is ​​obtained, including: Based on the similarity between the first local feature descriptor and the second local feature descriptor, a one-to-one correspondence is determined between multiple first key points included in the target region and multiple second key points included in the region to be matched; Based on the one-to-one correspondence, the Random Sampling Consensus (RANSAC) method is used to obtain the first transformation matrix.

6. The method according to claim 1, characterized in that, Based on the first transformation matrix, a third transformation matrix is ​​obtained, including: The first transformation matrix is ​​used to perform preliminary pose transformation on multiple first key points included in the target region to obtain an initial template point cloud; The iterative nearest point (ICP) method is used to find the minimum distance between the initial template point cloud and multiple second key points included in the region to be matched, and the second transformation matrix is ​​obtained. The first transformation matrix and the second transformation matrix are combined to obtain the third transformation matrix.

7. The method according to claim 1, characterized in that, The target region is determined from the 3D point cloud corresponding to the template image, including: From the 3D point cloud corresponding to the template image, select the region with significant appearance features as the target region; The regions with significant appearance features include high curvature regions and / or regions with local geometric structures; the high curvature regions include points with curvature exceeding a preset value; the local geometric structures include surfaces and / or corners with regular or irregular geometric shapes.

8. A vehicle exterior image registration device, characterized in that, include: The 3D point cloud processing module is used to determine the target region from the 3D point cloud corresponding to the template image; and to determine the region to be matched from the 3D point cloud corresponding to the image to be detected. The descriptor submodule is used to obtain a first local feature descriptor corresponding to a point in the target region and a second local feature descriptor corresponding to a point in the region to be matched; An initial transformation module is used to obtain a first transformation matrix based on the first local feature descriptor and the second local feature descriptor; A precise transformation module is used to obtain a third transformation matrix based on the first transformation matrix; The perspective transformation module is used to perform a two-dimensional perspective transformation on the third transformation matrix to obtain a pseudo-3D transformation matrix corresponding to the 2D image to be detected. The registration module is used to transform the 2D image to be detected using the pseudo-3D transformation matrix to obtain the registered image to be detected.

9. The apparatus according to claim 8, characterized in that, The descriptor module is specifically used to: obtain a first local feature descriptor corresponding to a point in the target region and a second local feature descriptor corresponding to a point in the region to be matched using the three-dimensional local feature descriptor SHOT.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the vehicle exterior image registration method as described in any one of claims 1 to 6.

Images