Image registration method and device, electronic equipment, readable storage medium and chip
By performing regional registration in infrared and visible light images and combining point and line features for joint optimization, the problem of large registration errors in existing technologies is solved, achieving higher fusion accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CSSC SYST ENG RES INST
- Filing Date
- 2024-12-26
- Publication Date
- 2026-07-03
Smart Images
Figure CN120047502B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically, to an image registration method, apparatus, electronic device, readable storage medium, and chip. Background Technology
[0002] Visible light images possess rich detail, but their performance is poor in harsh environments such as low light. Infrared images, on the other hand, provide better brightness information and thermal features under these conditions, but their detail is weaker. Therefore, effectively fusing visible light and infrared images has become an important means to improve battlefield target detection capabilities. Before fusing visible light and infrared images, registration of the two images is necessary to achieve accurate fusion. Currently, registration methods can be divided into three categories: feature-based registration methods, region-based registration methods, and phase-correlation-based registration methods. Traditional methods for registering infrared and visible light images have high requirements for image texture information, differences, and resolution, leading to significant registration errors. Summary of the Invention
[0003] In view of this, the present invention aims to solve the problem that the image registration process has high requirements for the texture information, differences and resolution of infrared and visible light images, which leads to large errors in the registration results.
[0004] Specifically, the present invention is achieved through the following technical solution:
[0005] The first aspect of this invention provides an image registration method.
[0006] A second aspect of the present invention provides an image registration apparatus.
[0007] A third aspect of the present invention provides an electronic device.
[0008] A fourth aspect of the present invention provides a readable storage medium.
[0009] The fifth aspect of the present invention provides a chip.
[0010] The image registration method provided by this invention includes: acquiring an original visible light image and an original infrared image of the same target scene; extracting features from the original visible light image and the original infrared image respectively to determine point features, the point features including a first point feature and a second point feature, the first point feature corresponding to the original visible light image and the second point feature corresponding to the original infrared image; extracting features from the original visible light image and the original infrared image respectively to determine line features, the line features including a first line feature and a second line feature, the first line feature corresponding to the original visible light image and the second line feature corresponding to the original infrared image; determining a registration region based on the point features and line features, the registration region including a first registration region and a second registration region, the first registration region corresponding to the original visible light image and the second registration region corresponding to the original infrared image; calculating the initial value of the registration matrix for the registration region to determine the initial value of the registration matrix; establishing an optimization equation based on the initial value of the registration matrix combined with the point features and line features to determine the registration result of the original visible light image and the original infrared image.
[0011] In some technical solutions, optionally, feature extraction is performed on the original visible light image and the original infrared image to determine point features, including: constructing Gaussian pyramids for the original visible light image and the original infrared image respectively; determining scale spaces, including a first scale space and a second scale space, the first scale space corresponding to the visible light image and the second scale space corresponding to the original infrared image; determining feature points in the scale space; determining feature descriptors based on the feature points; and determining point features in the original visible light image and the infrared image based on the feature descriptors.
[0012] In some technical solutions, optionally, feature extraction is performed on the original visible light image and the original infrared image to determine line features, including: scaling the original visible light image and the original infrared image respectively to determine a first original visible light image and a first original infrared image; calculating gradients on the first original visible light image and the first original infrared image respectively to determine gradient values, the gradient values including visible light gradient values and infrared gradient values; determining line segment support regions corresponding to the first original visible light image and the first original infrared image based on the gradient values; performing rectangularization processing on the line segment support regions, and determining the line features in the original visible light image and the infrared image based on the rectangles obtained after processing.
[0013] In some technical solutions, optionally, the registration region is determined based on line features and line features, including: acquiring a historical image database; determining a point feature density threshold and a line feature density threshold based on the historical image database; determining a first point feature density based on at least one first point feature; when the value of the first point feature density is less than the point feature density threshold, determining the region corresponding to the first point feature density as the first registration region; determining a first line feature density based on at least one first line feature; when the value of the first line feature density is less than the line feature density threshold, determining the region corresponding to the first line feature density as the first registration region; and determining at least one first registration region corresponding to the original visible light image.
[0014] In some technical solutions, optionally, determining the registration region based on point features and line features further includes: determining a second point feature density based on at least one second point feature; when the value of the second point feature density is less than a point feature density threshold, determining the region corresponding to the second point feature density as a second registration region; determining a second line feature density based on at least one second line feature; when the value of the second line feature density is less than a line feature density threshold, determining the region corresponding to the second line feature density as a second registration region; and determining at least one second registration region corresponding to the original infrared image.
[0015] In some technical solutions, optionally, determining the registration region based on point features and line features further includes: determining the point feature density corresponding to at least one point feature; when the point feature density is greater than a point feature density threshold, determining the region corresponding to at least one point feature as a point-line feature registration region; determining the line feature density corresponding to at least one line feature; when the line feature density is greater than a line feature density threshold, determining the region corresponding to at least one line feature as a point-line feature registration region, wherein the point-line feature registration region includes a first point-line feature registration region and a second point-line feature registration region, the first point-line feature registration region corresponding to the original visible light image, and the second point-line feature registration region corresponding to the original infrared image.
[0016] A second aspect of the present invention provides an image registration apparatus, comprising: an acquisition module for acquiring an original visible light image and an original infrared image of the same target scene; a point feature extraction module for extracting features from the original visible light image and the original infrared image respectively to determine point features; a line feature extraction module for extracting features from the original visible light image and the original infrared image respectively to determine line features; a region determination module for determining a registration region based on the point features and line features; an initial value calculation module for performing initial value calculation of the registration matrix on the registration region to determine the initial value of the registration matrix; and a registration module for establishing an optimization equation based on the initial value of the registration matrix combined with the point features and line features to determine the registration result of the original visible light image and the original infrared image.
[0017] An embodiment of the third aspect of the present invention provides an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps as described in the first aspect.
[0018] An embodiment of the fourth aspect of the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps as described in the first aspect.
[0019] An embodiment of the fifth aspect of the present invention provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run a program or instructions to implement the steps as described in the first aspect.
[0020] The technical solution provided by this invention brings at least the following beneficial effects:
[0021] This invention proposes an image registration method that performs regional registration on infrared and visible light images acquired from the same scene, and performs joint optimization by combining point feature line feature relationships, outputting a registration matrix and registration results, thereby improving the robustness and accuracy of infrared and visible light image fusion registration. Attached Figure Description
[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0024] Figure 1 This is a schematic flowchart of the image registration method provided in an embodiment of the present invention;
[0025] Figure 2 This is a partial flowchart illustrating the image registration method provided in an embodiment of the present invention;
[0026] Figure 3 This is a partial flowchart illustrating the image registration method provided in an embodiment of the present invention;
[0027] Figure 4 This is a partial flowchart illustrating the image registration method provided in an embodiment of the present invention;
[0028] Figure 5 This is a partial flowchart illustrating the image registration method provided in an embodiment of the present invention;
[0029] Figure 6 This is a partial flowchart illustrating the image registration method provided in an embodiment of the present invention;
[0030] Figure 7 This is a schematic block diagram of the image registration device provided in an embodiment of the present invention;
[0031] Figure 8 A schematic block diagram of the structure of an electronic device provided in an embodiment of the present invention;
[0032] Figure 9 This is a schematic flowchart of the image registration method provided in an embodiment of the present invention.
[0033] in, Figure 7 and Figure 8 The correspondence between component names and their designations is as follows:
[0034] 900: Image registration device; 902: Acquisition module; 904: Point feature extraction module; 906: Line feature extraction module; 908: Region determination module; 910: Initial value calculation module; 912: Registration module; 1000: Electronic device; 1109: Memory; 1110: Processor. Detailed Implementation
[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] See Figure 1 The first aspect of the present invention provides an image registration method, comprising the following steps:
[0037] Step S100: Acquire the original visible light image and the original infrared image of the same target scene;
[0038] Step S102: Extract features from the original visible light image and the original infrared image respectively, and determine the point features. The point features include the first point feature and the second point feature. The first point feature corresponds to the original visible light image, and the second point feature corresponds to the original infrared image.
[0039] Step S104: Extract features from the original visible light image and the original infrared image respectively, and determine the line features. The line features include a first line feature and a second line feature. The first line feature corresponds to the original visible light image, and the second line feature corresponds to the original infrared image.
[0040] Step S106: Determine the registration region based on point features and line features. The registration region includes a first registration region and a second registration region. The first registration region corresponds to the original visible light image, and the second registration region corresponds to the original infrared image.
[0041] Step S108: Perform initial value calculation of the registration matrix for the registration area to determine the initial value of the registration matrix;
[0042] Step S110: Based on the initial value of the registration matrix and the point and line features, establish an optimization equation to determine the registration results of the original visible light image and the original infrared image.
[0043] According to the image registration method provided by this invention, infrared and visible light images acquired from the same scene are registered in different regions to determine feature registration regions. Point and line features are extracted from the feature registration regions of the infrared and visible light images respectively. The regions of the infrared and visible light images are divided to determine the registration regions. Point-line feature associations are established using the spatial positional relationship of point and line features as point-line association features. Finally, the region registration result is calculated and used as an initial value. Joint optimization is performed using point features, line features, and point-line association features to obtain the registration result of the infrared and visible light images. Specifically, visible light and infrared images from the same scene are acquired by visible light and infrared sensors and used as the original visible light and original infrared images. Feature extraction is performed on both the original visible light and infrared images to extract common features, namely point features and line segment features. Based on the feature matching results, data association and matching relationships between the infrared and visible light images are established. Specifically, point feature extraction is performed on both images to determine feature descriptors for key points in each image. At least one point feature (the second point feature) is determined based on the feature descriptor of the infrared image, and at least one point feature (the first point feature) is determined based on the feature descriptor of the visible light image. After point feature extraction, line feature extraction is performed on both images. This is done by defining the line segment support regions in the images and rectangularizing these regions to obtain relatively regular rectangular areas as the line segment extraction results. Line segments corresponding to these rectangular areas in both the infrared and visible light images are then determined, thus identifying at least one line feature (the second line feature) for the infrared image and at least one line feature (the first line feature) for the visible light image. Feature association is performed based on the extracted point and line features of the infrared and visible light images. Feature regions of the infrared and visible light images are analyzed by setting point and line feature densities respectively. High-feature regions (areas with point and line feature densities higher than corresponding thresholds) are identified based on these densities, indicating strong data correlation; this is used for feature-based registration. Low-feature regions (areas with point and line feature densities lower than corresponding thresholds) are also identified based on these densities; these are the registration regions. The similarity between the registration regions of the visible light and infrared images is determined using the statistical characteristics of pixel grayscale values.Specifically, the initial value of the registration matrix is calculated for the visible light image and the infrared image through the registration region. An optimization equation is constructed. After the initial value is calculated, the optimization equation is constructed using the point features of the infrared image, the point features of the visible light image, the line features of the infrared image, and the line features of the visible light image. The region registration problem is transformed into a least squares problem. The initial value of the registration matrix is used as the initial value of the least squares optimization iteration to determine the registration result of the infrared image and the visible light image.
[0044] Understandably, by extracting point and line features from infrared and visible light images respectively, and determining the registration regions corresponding to the infrared and visible light images based on their respective point and line features, the region-based registration method and the feature-based registration method are combined. The results of region registration are used as the initial values for point and line feature registration, which improves the accuracy of the registration matrix calculation results and thus improves the fusion accuracy between visible light and infrared images.
[0045] In some embodiments, optionally, such as Figure 2 As shown, feature extraction is performed on the original visible light image and the original infrared image respectively, and the point features are determined as follows:
[0046] Step S1022: Construct Gaussian pyramids for the original visible light image and the original infrared image respectively, and determine the scale space. The scale space includes a first scale space and a second scale space. The first scale space corresponds to the original visible light image, and the second scale space corresponds to the original infrared image.
[0047] Step S1024: Determine feature points in scale space;
[0048] Step S1026: Determine the feature descriptor based on the feature points;
[0049] Step S1028: Determine the point features in the original visible light image and infrared image based on the feature descriptors.
[0050] In this embodiment, point features are extracted from the acquired original visible light image and the original infrared image, respectively. Specifically, since the infrared texture features in the infrared image are limited, i.e., the texture information of the original infrared image is sparse, the Scale Invariant Feature Transform (SIFT) method is selected to process the original visible light image and the original infrared image. The corner points of the image, i.e., the locations of points with local maximum curvature or significant gradient changes, will not be easily changed by factors such as illumination, radiometric transformation, and noise. First, Gaussian pyramids are constructed for the original visible light image and the original infrared image, respectively. Gaussian blurring is performed on each layer using different parameters, thereby constructing the Difference of Gaussians (DG) model. The Gaussian (DOG) scale space is defined as follows: the first scale space corresponds to the scale space used to construct the Gaussian pyramid from the original visible light image, and the second scale space corresponds to the scale space used to construct the Gaussian pyramid from the original infrared image. Feature points corresponding to the original visible light image in the first scale space and the original infrared image in the second scale space are determined. Feature points are local extrema with directional information detected in images at different scale spaces. These extrema do not disappear due to changes in lighting conditions; examples include corner points, edge points, bright spots in dark areas, and dark spots in bright areas. Since the two images contain the same objects, these feature points will have corresponding matches. Based on feature points in the first scale space and feature points in the second scale space, feature descriptors corresponding to the original visible light image and the original infrared image are determined respectively. The feature descriptors are used to determine the point features in the original visible light image and the original infrared image. They are obtained by extracting feature points in the first scale space and the second scale space. The feature descriptors have discriminativeness, invariance, robustness and compactness. The method of determining the point features of the original visible light image and the original infrared image by using feature descriptors makes the feature points easy to distinguish and remain stable, and uses as few dimensions or information as possible to describe the feature information around the feature points, so as to improve computational efficiency and storage efficiency.
[0051] In some embodiments, optionally, such as Figure 3 As shown, feature extraction is performed on the original visible light image and the original infrared image respectively, and the line features are determined to include:
[0052] Step S1042: Scale the original visible light image and the original infrared image respectively to determine the first original visible light image and the first original infrared image;
[0053] Step S1044: Perform gradient calculations on the first original visible light image and the first original infrared image respectively, and determine the gradient values, including the visible light gradient value and the infrared gradient value.
[0054] Step S1046: Determine the line segment support region corresponding to the first original visible light image and the first original infrared image based on the gradient value;
[0055] Step S1048: Rectify the line segment support region and determine the line features in the original visible light image and infrared image based on the rectangle obtained after processing.
[0056] In this embodiment, line features are extracted from the original visible light image and the original infrared image respectively. The Line Segment Detector (LSD) algorithm is used to detect line segments in both images. This method can obtain high-precision line segment detection results in a short time. First, the gradient magnitude and direction of all points in the image are calculated. Then, adjacent points with small gradient direction changes are considered as a connected region. Based on the rectangularity of each region, it is determined whether it needs to be broken up according to rules to form multiple regions with larger rectangularities. Finally, all generated regions are improved and filtered, and the regions that meet the conditions are retained, which is the final line detection result. Specifically, the original visible light image and the original infrared image are scaled respectively, and the scaled images are determined as the first original visible light image and the first original infrared image, respectively. Gradient calculations are performed on the first original visible light image and the first original infrared image to determine the visible light gradient value corresponding to the first original visible light image and the infrared gradient value corresponding to the first original infrared image. The method for calculating the gradient values includes: calculating the gradient of the four pixels to the right of each pixel in the image. Assuming the gray level of pixel (x,y) is i(x,y), then the pixel gradient g of that pixel on the x-axis and y-axis is... x (x, y) and g y (x, y) are respectively:
[0057]
[0058] Where x is the x-coordinate of the pixel on the x-axis and y is the y-coordinate of the pixel on the y-axis.
[0059] Based on the pixel gradients along the x and y axes, we can obtain the gradient magnitude G(x, y) and gradient direction LLA:
[0060]
[0061] After determining the pixel gradient, the gradients are sorted. The larger the gradient magnitude calculated for a pixel in the image, the more significant the edge point. A threshold for gradient magnitude is set for filtering, selecting points with larger magnitudes. Then, considering that there are many supporting pixels around the line segment support region, an isolated pixel is randomly selected from the sorted list. The directional tolerance value between the gradient direction of the isolated pixel and the gradient direction of the support region is calculated to see if it is less than r. If r meets the threshold, the isolated pixel is changed to USED and included in the line segment support region, and the line segment support region after adding the isolated pixel is updated. Here, r is the set directional error tolerance value, representing the error value between the rectangular direction of the line segment support region and the pixel. When r < 22.5, the pixel will be included in the original line segment support region. Next, the updated line segment support region is rectangularized, and a rectangular approximation calculation is performed on the line segment support region to obtain a more regular rectangular region as the line segment extraction result for display. The line segment support regions corresponding to the first original visible light image and the first original infrared image are determined respectively. By setting the pixel density value f in the rectangular region, each generated region is judged. When the number of pixels in the region supported by the rectangular line segment is greater than the pixel density value f, the corresponding line segment is determined to be a line feature in the first original visible light image or the first original infrared image.
[0062] In some embodiments, optionally, such as Figure 4 As shown, the registration region is determined based on line features and line characteristics, including:
[0063] Step S1062: Obtain the historical image database, and determine the point feature density threshold and line feature density threshold based on the historical image database;
[0064] Step S1064: Determine the first point feature density based on at least one first point feature;
[0065] Step S1066: When the value of the first point feature density is less than the point feature density threshold, the region corresponding to the first point feature density is determined as the first registration region;
[0066] Step S1068: Determine the first line feature density based on at least one first line feature;
[0067] Step S1070: When the value of the first line feature density is less than the line feature density threshold, the region corresponding to the first line feature density is determined as the first registration region;
[0068] Step S1072: Determine at least one first registration region corresponding to the original visible light image.
[0069] In this embodiment, the operator analyzes images in the historical image database and determines the point feature density threshold k and the line feature density threshold m based on the point feature density and line feature density in the historical image data. The point feature density threshold k indicates that when the point feature density of a certain region in the image is k, that region can be used as a feature region in the image registration process, indicating strong feature characteristics. The line feature density m indicates that when the line feature density of a certain region in the image is m, that region can be used as a feature region in the image registration process, indicating strong feature characteristics. Because in practical applications, the image data in the historical image database will retain historical registration thresholds with good registration results, the values of m and k can be taken as the average of at least one historical registration threshold. The process involves several steps: First, a first point feature corresponding to the original visible light image is identified. When the density of this first point feature in the original visible light image is less than a point feature density threshold k, the region corresponding to this first point feature density is considered to have weak eigenvalues, i.e., sparse image texture. This region can be used for registration using region registration methods, and is thus designated as the first registration region. Second, a first line feature corresponding to the original visible light image is identified. When the density of this first line feature in the original visible light image is less than a point feature density threshold m, the region corresponding to this first line feature density is considered to have weak eigenvalues, i.e., sparse image texture. This region can also be used for registration using region registration methods, and is thus designated as the first registration region. The region registration method involves first using one image as a template and searching for the region in another image that is most similar to the template. Second, the statistical characteristics of pixel grayscale values are used to evaluate the similarity between the images. Finally, the transformation parameters between the images are adjusted to maximize the similarity index, thereby achieving registration.
[0070] In some embodiments, optionally, such as Figure 5 As shown, determining the registration area based on point and line features also includes:
[0071] Step S1162: Determine the second point feature density based on at least one second point feature;
[0072] Step S1164: When the value of the second point feature density is less than the point feature density threshold, the region corresponding to the second point feature density is determined as the second registration region;
[0073] Step S1166: Determine the density of the second line feature based on at least one second line feature;
[0074] Step S1168: When the value of the second line feature density is less than the line feature density threshold, the region corresponding to the second line feature density is determined as the second registration region;
[0075] Step S1170: Determine at least one second registration region corresponding to the original infrared image.
[0076] In this embodiment, a second point feature corresponding to the original infrared image is determined. When the density of the second point feature in the original infrared image is less than the point feature density threshold k, the region corresponding to the second point feature density is determined to have weak feature characteristics, i.e., sparse image texture. This region can be used for registration using a region registration method, i.e., the second registration region. A second line feature corresponding to the original infrared image is also determined. When the density of the second line feature in the original infrared image is less than the point feature density threshold m, the region corresponding to the second line feature density is determined to have weak feature characteristics, i.e., sparse image texture. This region can be used for registration using a region registration method, i.e., the second registration region.
[0077] In some embodiments, optionally, such as Figure 6 As shown, determining the registration area based on point and line features also includes:
[0078] Step S1262: Determine the point feature density corresponding to at least one point feature;
[0079] Step S1264: When the point feature density is greater than the point feature density threshold, determine at least one region corresponding to a point feature as the point-line feature registration region;
[0080] Step S1266: Determine the line feature density corresponding to at least one line feature;
[0081] Step S1268: When the line feature density is greater than the line feature density threshold, determine at least one region corresponding to a line feature as a point-line feature registration region. The point-line feature registration region includes a first point-line feature registration region and a second point-line feature registration region.
[0082] In this embodiment, when the first point feature density corresponding to the original visible light image is greater than the point feature density threshold k, the region corresponding to the first point feature density is determined to have strong eigenvalues, and this region can be used for point-line feature registration. Similarly, when the first line feature density corresponding to the original visible light image is greater than the line feature density threshold m, the region corresponding to the first line feature density is determined to have strong eigenvalues, and this region can be used for point-line feature registration. The regions corresponding to the first point feature density and the first line feature density are the first point-line feature registration regions. Likewise, when the second point feature density corresponding to the original infrared image is greater than the point feature density threshold m, the region corresponding to the second point feature density is determined to have strong eigenvalues, and this region can be used for point-line feature registration. Similarly, when the second line feature density corresponding to the original infrared image is greater than the line feature density threshold k, the region corresponding to the second line feature density is determined to have strong eigenvalues, and this region can be used for point-line feature registration. The regions corresponding to the second point feature density and the second line feature density are the second point-line feature registration regions. This region registration method, combining point and line features, improves the data correlation between the original visible light image and the original infrared image registration based on the constraint relationship of feature points or feature lines.
[0083] In a specific embodiment, the image registration method mainly includes point-line association feature extraction, point-line association feature matching, region matching, and feature-region joint matching, such as... Figure 9 As shown, the image registration method includes: Step S800: Acquire a visible light image; Step S900: Acquire an infrared image; Step S802: Extract the first point feature; Step S902: Extract the second point feature; Step S806: Extract the first line feature; Step S906: Extract the second line feature; Step S804: Divide the first region; Step S904: Divide the second region; Step S808: Extract the first region feature; Step S908: Extract the second region feature; Step S810: Pair the first point and line features; Step S910: Pair the second point and line features; Step S988: Initialize the registration matrix; Step S999: Jointly optimize multiple features of the visible light image and the infrared image.
[0084] In this embodiment, firstly, the infrared image and the visible light image are divided into regions to define the feature registration region and the region registration region. Secondly, point features and line features are extracted from the feature registration regions of the infrared image and the visible light image respectively, and the region registration regions of the infrared image and the visible light image are defined. Point-line feature association is established using the spatial positional relationship of point features and line features as point-line association features. Finally, the region registration result is calculated and used as the initial value. Point features, line features, and point-line association features are used for joint optimization to obtain the registration result of the infrared image and the visible light image.
[0085] Specifically, for point feature extraction, considering the limited infrared texture features, this invention chooses SIFT (Scale-Invariant Feature Transform), as corner points are not easily altered by factors such as illumination, radiometric transformation, and noise. First, to simulate the multi-scale features of image data, a Gaussian pyramid is constructed, with different parameters applied to each layer for Gaussian blurring, thus constructing a Difference of Gaussian (DOG) scale space. Second, in the constructed DOG scale space, extreme points are searched as candidate feature points. If a point has the maximum or minimum value in the 26 neighborhoods of the layers above and below it in the DOG scale space, it is considered a feature point of the image at that scale. Then, the gradient direction histogram of each keypoint is calculated, and vectors claiming uniqueness are combined to form the feature descriptor of the keypoint. At this point, feature point extraction is complete.
[0086] For line feature extraction, this invention uses LSD (Line Segment Detector), a method that can obtain high-precision line detection results in a short time. First, the image is scaled, and the LSD gradient is calculated. This involves calculating the gradient of the four pixels to the right of each pixel in the image. Assuming the gray level of pixel (x, y) is i(x, y), then the pixel gradient g of that pixel along the x-axis and y-axis is calculated. x (x, y) and g y (x, y) are respectively:
[0087]
[0088] Where x is the x-coordinate of the pixel on the x-axis and y is the y-coordinate of the pixel on the y-axis.
[0089] Based on the pixel gradients along the x and y axes, we can obtain the gradient magnitude G(x, y) and gradient direction LLA:
[0090]
[0091] The gradients are sorted, with larger gradient magnitudes indicating more significant edge points and making them more suitable as seed points for line segment detection. A threshold for gradient magnitude is set to filter and select points with larger magnitudes. Then, considering the large number of supporting pixels around the line segment support domain, an isolated pixel is randomly selected from the sorted list. The directional tolerance between the gradient direction of the isolated pixel and the gradient direction of the support domain is calculated to see if it is less than r. If r meets the threshold, the isolated pixel is changed to a used state and included in the line segment support domain, updating the line segment support domain after adding the isolated pixel. Here, r is the set directional error tolerance value, representing the error value between the rectangular direction of the line segment support domain and the pixel. When r < 22.5, all pixels are included in the original line segment support region. Next, the updated line segment support domain is rectangularized by performing a rectangular approximation calculation on the line segment support domain region to obtain a more regular rectangular region as the line segment extraction result.
[0092]
[0093] Where τ represents the gradient angle of all points obtained by LLA in the region surrounding the pixel, l x Let l be the x-coordinate of the center of the rectangle. y Let G(j) be the y-coordinate of the center of the rectangle, and G(j) be the gradient magnitude of pixel j. j∈Rejion G(j) represents all pixels within the convenience support domain of the line segment, and (x(j), y(j)) represents the coordinates of a point in the region. The principal direction M of the rectangular region after the line segment support domain is rectangularized is:
[0094]
[0095] in,
[0096]
[0097] Finally, estimating the number of pixels in the rectangle that satisfy the gradient magnitude can be used to determine whether the rectangle can be a "line segment". The more pixels that satisfy the pixel gradient magnitude, the more likely the rectangle is to be a "line segment". Each generated rectangle is judged by setting the pixel density value f in the rectangle.
[0098] Image region segmentation: Based on the point and line features calculated above, the image is segmented. Appropriate thresholds k and m are set. When the point feature density in the infrared image or the visible light image is less than k or the line feature density is less than m, region registration is performed on the region; otherwise, point and line feature registration is performed.
[0099] Multi-feature region joint registration optimization involves initial value calculation of the registration matrix for each region to be registered, and construction of the optimization equation:
[0100]
[0101] Where T0 is the initial value of the registration matrix, I(x, y) is the gray value of the pixel at coordinates (x, y) in the infrared image, and V(x, y) is the gray value of the pixel at coordinates (x, y) in the visible light image. The region registration problem is transformed into a least-squares problem, thus obtaining the initial value of the registration matrix. After calculating the initial value, the following optimization equation is constructed using point and line features:
[0102]
[0103] Where T iv Let P be the registration matrix. I (z) represents the coordinates of the z-feature point in the infrared image, P v (z) represents the coordinates of the z-axis feature point in the visible light image, L I (w) represents the coordinates of the w-line feature in the infrared image, L V (w) represents the coordinates of the w line in the visible light image. Similarly, the region registration problem is transformed into a least squares problem, with T0 as the initial value for the least squares optimization iteration.
[0104] like Figure 7As shown, a second aspect of the present invention provides an image registration device 900, comprising: an acquisition module 902 for acquiring an original visible light image and an original infrared image of the same target scene; a point feature extraction module 904 for extracting features from the original visible light image and the original infrared image respectively to determine point features; a line feature extraction module 906 for extracting features from the original visible light image and the original infrared image respectively to determine line features; a region determination module 908 for determining a registration region based on the point features and line features; an initial value calculation module 910 for performing initial value calculation of the registration matrix on the registration region to determine the initial value of the registration matrix; and a registration module 912 for establishing an optimization equation based on the initial value of the registration matrix combined with the point features and line features to determine the registration result of the original visible light image and the original infrared image.
[0105] like Figure 8 As shown, the third aspect of the present invention provides an electronic device 1000, including a processor 1110, a memory 1109, and a program or instructions stored in the memory 1109 and executable on the processor 1110. When the program or instructions are executed by the processor 1110, they implement the various processes of the above-described image registration method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0106] The processor 1110 is used to acquire the original visible light image and the original infrared image of the same target scene; extract features from the original visible light image and the original infrared image to determine point features; extract features from the original visible light image and the original infrared image to determine line features; determine the registration area based on the point features and line features; calculate the initial value of the registration matrix for the registration area to determine the initial value of the registration matrix; and establish an optimization equation based on the initial value of the registration matrix combined with the point features and line features to determine the registration result of the original visible light image and the original infrared image.
[0107] Optionally, the processor 1110 is also configured to construct Gaussian pyramids for the original visible light image and the original infrared image respectively, determine the scale space; determine feature points in the scale space; determine feature descriptors based on the feature points; and determine point features in the original visible light image and the infrared image based on the feature descriptors.
[0108] Optionally, the processor 1110 is further configured to scale the original visible light image and the original infrared image respectively to determine the first original visible light image and the first original infrared image; perform gradient calculation on the first original visible light image and the first original infrared image respectively to determine the gradient value, the gradient value including the visible light gradient value and the infrared gradient value; determine the line segment support region corresponding to the first original visible light image and the first original infrared image according to the gradient value; perform rectangular processing on the line segment support region, and determine the line features in the original visible light image and the infrared image according to the rectangle obtained after processing.
[0109] Optionally, the processor 1110 is further configured to acquire a historical image database, determine a point feature density threshold and a line feature density threshold based on the historical image database; determine a first point feature density based on at least one first point feature; when the value of the first point feature density is less than the point feature density threshold, determine the region corresponding to the first point feature density as a first registration region; determine a first line feature density based on at least one first line feature; when the value of the first line feature density is less than the line feature density threshold, determine the region corresponding to the first line feature density as a first registration region; and determine at least one first registration region corresponding to the original visible light image.
[0110] Optionally, the processor 1110 is further configured to: determine a second point feature density based on at least one second point feature; determine the region corresponding to the second point feature density as a second registration region when the value of the second point feature density is less than a point feature density threshold; determine a second line feature density based on at least one second line feature; determine the region corresponding to the second line feature density as a second registration region when the value of the second line feature density is less than a line feature density threshold; and determine at least one second registration region corresponding to the original infrared image.
[0111] Optionally, the processor 1110 is further configured to determine the point feature density corresponding to at least one point feature; when the point feature density is greater than a point feature density threshold, determine the region corresponding to at least one point feature as a point-line feature registration region; determine the line feature density corresponding to at least one line feature; when the line feature density is greater than a line feature density threshold, determine the region corresponding to at least one line feature as a point-line feature registration region.
[0112] A fourth aspect of this invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the various processes of the embodiments of the image registration method described above, achieving the same technical effects. To avoid repetition, these will not be elaborated further here. Furthermore, the readable storage medium improves the data storage capacity and data processing speed of the image registration method in this application.
[0113] The methods can be implemented in various ways depending on the specific features and / or example applications. For example, these methods can be implemented by a combination of hardware, firmware, and / or software. For instance, in a hardware implementation, the processor can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other device units for performing the functions described above, and / or combinations thereof.
[0114] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, but is not limited thereto. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital universal disk (DVD), memory cards, floppy disks, encoding mechanical devices (e.g., punched cards or grooves with raised structures for recording instructions), and any suitable combination of the foregoing. The computer-readable storage medium used herein should not be construed as the transmission of signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media, or electrical signals transmitted through wires.
[0115] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.
[0116] A fifth aspect of this invention provides a chip including a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the embodiments of the above-described image registration method, achieving the same technical effects. To avoid repetition, these will not be described again here. Furthermore, the chip improves the data processing speed corresponding to the image registration method in this application.
[0117] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily used to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.
[0118] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0119] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.
[0120] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0121] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. An image registration method, characterized in that, include: Acquire the original visible light image and the original infrared image of the same target scene; Feature extraction is performed on the original visible light image and the original infrared image respectively to determine point features. The point features include a first point feature and a second point feature. The first point feature corresponds to the original visible light image, and the second point feature corresponds to the original infrared image. Feature extraction is performed on the original visible light image and the original infrared image respectively to determine line features. The line features include a first line feature and a second line feature. The first line feature corresponds to the original visible light image, and the second line feature corresponds to the original infrared image. The registration region is determined based on the point features and the line features. The registration region includes a first registration region and a second registration region. The first registration region corresponds to the original visible light image, and the second registration region corresponds to the original infrared image. The initial value of the registration matrix is calculated for the registration region to determine the initial value of the registration matrix; An optimization equation is established based on the initial value of the registration matrix, combined with the point features and the line features, to determine the registration result of the original visible light image and the original infrared image; Determining the registration region based on the point features and the line features includes: Obtain a historical image database, and determine point feature density thresholds and line feature density thresholds based on the historical image database; The first point feature density is determined based on at least one of the first point features; When the value of the first point feature density is less than the point feature density threshold, the region corresponding to the first point feature density is determined as the first registration region. The density of the first line feature is determined based on at least one of the first line features; When the value of the first line feature density is less than the line feature density threshold, the region corresponding to the first line feature density is determined as the first registration region. Determine at least one first registration region corresponding to the original visible light image; The step of determining the registration region based on the point features and the line features further includes: The second point feature density is determined based on at least one of the second point features; When the value of the second point feature density is less than the point feature density threshold, the region corresponding to the second point feature density is determined as the second registration region; The density of the second line feature is determined based on at least one of the second line features; When the value of the second line feature density is less than the line feature density threshold, the region corresponding to the second line feature density is determined as the second registration region; Determine at least one second registration region corresponding to the original infrared image.
2. The image registration method according to claim 1, characterized in that, The step of extracting features from the original visible light image and the original infrared image respectively to determine point features includes: Gaussian pyramids are constructed for the original visible light image and the original infrared image respectively to determine the scale space. The scale space includes a first scale space and a second scale space. The first scale space corresponds to the original visible light image, and the second scale space corresponds to the original infrared image. Determine the feature points in the scale space; Determine the feature descriptor based on the feature points; Point features in the original visible light image and the infrared image are determined based on the feature descriptor.
3. The image registration method according to claim 1, characterized in that, The step of extracting features from the original visible light image and the original infrared image respectively to determine line features includes: The original visible light image and the original infrared image are scaled respectively to determine the first original visible light image and the first original infrared image; Gradient calculations are performed on the first original visible light image and the first original infrared image respectively to determine gradient values, which include visible light gradient values and infrared gradient values. The line segment support region corresponding to the first original visible light image and the first original infrared image is determined based on the gradient value; The line segment support region is rectangularized, and the line features in the original visible light image and the infrared image are determined based on the rectangle obtained after processing.
4. The image registration method according to claim 1, characterized in that, The step of determining the registration region based on the point features and the line features further includes: Determine the point feature density corresponding to at least one of the point features; When the point feature density is greater than the point feature density threshold, at least one region corresponding to the point feature is determined as a point-line feature registration region. Determine the line feature density corresponding to at least one of the line features; When the line feature density is greater than the line feature density threshold, at least one region corresponding to the line feature is determined as a point-line feature registration region. The point-line feature registration region includes a first point-line feature registration region and a second point-line feature registration region. The first point-line feature registration region corresponds to the original visible light image, and the second point-line feature registration region corresponds to the original infrared image.
5. An image registration device, characterized in that, include: The acquisition module is used to acquire the original visible light image and the original infrared image of the same target scene; The point feature extraction module is used to extract features from the original visible light image and the original infrared image respectively to determine point features; the point features include a first point feature and a second point feature, wherein the first point feature corresponds to the original visible light image and the second point feature corresponds to the original infrared image; A line feature extraction module is used to extract features from the original visible light image and the original infrared image respectively to determine line features; the line features include a first line feature and a second line feature, wherein the first line feature corresponds to the original visible light image and the second line feature corresponds to the original infrared image; A region determination module is used to determine a registration region based on the point features and the line features; the registration region includes a first registration region and a second registration region, the first registration region corresponding to the original visible light image and the second registration region corresponding to the original infrared image; The initial value calculation module is used to perform initial value calculation of the registration matrix for the registration region and determine the initial value of the registration matrix; The registration module is used to establish an optimization equation based on the initial value of the registration matrix, the point features, and the line features, and to determine the registration result of the original visible light image and the original infrared image. Determining the registration region based on the point features and the line features includes: Obtain a historical image database, and determine point feature density thresholds and line feature density thresholds based on the historical image database; The first point feature density is determined based on at least one of the first point features; When the value of the first point feature density is less than the point feature density threshold, the region corresponding to the first point feature density is determined as the first registration region. The density of the first line feature is determined based on at least one of the first line features; When the value of the first line feature density is less than the line feature density threshold, the region corresponding to the first line feature density is determined as the first registration region. Determine at least one first registration region corresponding to the original visible light image; The step of determining the registration region based on the point features and the line features further includes: The second point feature density is determined based on at least one of the second point features; When the value of the second point feature density is less than the point feature density threshold, the region corresponding to the second point feature density is determined as the second registration region; The density of the second line feature is determined based on at least one of the second line features; When the value of the second line feature density is less than the line feature density threshold, the region corresponding to the second line feature density is determined as the second registration region; Determine at least one second registration region corresponding to the original infrared image.
6. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in any one of claims 1 to 4.
7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 4.
8. A chip, characterized in that, The chip includes a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method as described in any one of claims 1 to 4.