Image registration method and device, electronic equipment, readable storage medium and chip

Abstract

The invention provides an image registration method and device, electronic equipment, a readable storage medium and a chip, and the method comprises the steps: obtaining an original visible light image and an original infrared image in a same target scene; feature extraction is carried out on the original visible light image and the original infrared image, and point features are determined; feature extraction is carried out on the original visible light image and the original infrared image, and line features are determined; determining a registration area according to the point features and the line features; performing initial value calculation of a registration matrix on the registration region, and determining an initial value of the registration matrix; and establishing an optimization equation according to the registration matrix initial value combination point features and the line features, and determining a registration result of the original visible light image and the original infrared image. Through the scheme provided by the invention, region division is performed on the image, joint optimization is performed in combination with the point feature line feature relationship, the registration matrix and the registration result are output, and the robustness and accuracy of fusion registration of the infrared image and the visible light image are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image registration method, device, electronic device, readable storage medium and chip. Background Art

[0002] Visible light images have rich detail information, but they are not effective in harsh environments such as low light; while infrared images can provide better brightness information and thermal features under these conditions, but the details are weak. Therefore, the effective fusion of visible light images and infrared images has become an important means to improve the ability to detect battlefield targets. Before fusing visible light images and infrared images, the two images need to be registered to achieve accurate fusion of the two. At present, registration methods can be divided into three categories: feature-based registration methods, region-based registration methods, and phase-correlation-based registration methods. Traditional registration methods for infrared images and visible light images have high requirements on image texture information, differences, and resolution, resulting in large errors in registration results. Summary of the invention

[0003] In view of this, the present invention aims to solve the problem that the image registration process has high requirements on the texture information, difference and resolution of infrared images and visible light images, resulting in large errors in the registration results.

[0004] Specifically, the present invention is achieved through the following technical solutions:

[0005] A first aspect of the present invention provides an image registration method.

[0006] A second aspect of the present invention provides an image registration device.

[0007] A third aspect of the present invention provides an electronic device.

[0008] A fourth aspect of the present invention provides a readable storage medium.

[0009] A fifth aspect of the present invention provides a chip.

[0010] The image registration method provided by the present invention comprises: acquiring an original visible light image and an original infrared image under the same target scene; performing feature extraction on the original visible light image and the original infrared image respectively to determine point features, wherein the point features include a first point feature and a second point feature, wherein the first point feature corresponds to the original visible light image, and the second point feature corresponds to the original infrared image; performing feature extraction on the original visible light image and the original infrared image respectively to determine line features, wherein the line features include a first line feature and a second line feature, wherein the first line feature corresponds to the original visible light image, and the second line feature corresponds to the original infrared image; determining a registration area according to the point features and the line features, wherein the registration area includes a first registration area and a second registration area, wherein the first registration area corresponds to the original visible light image, and the second registration area corresponds to the original infrared image; performing initial value solution of a registration matrix on the registration area to determine an initial value of the registration matrix; establishing an optimization equation according to the initial value of the registration matrix in combination with the point features and the line features to determine the registration result of the original visible light image and the original infrared image.

[0011] In some technical solutions, optionally, feature extraction is performed on the original visible light image and the original infrared image respectively to determine point features, including: constructing Gaussian pyramids for the original visible light image and the original infrared image respectively, determining a scale space, the scale space including a first scale space and a second scale space, the first scale space corresponds to the visible light image, and the second scale space corresponds to the original infrared image; determining feature points in the scale space; determining feature descriptors based on the feature points; and determining point features in the original visible light image and the infrared image based on the feature descriptors.

[0012] In some technical schemes, optionally, feature extraction is performed on the original visible light image and the original infrared image respectively to determine line features, including: scaling the original visible light image and the original infrared image respectively to determine a first original visible light image and a first original infrared image; performing gradient calculation on the first original visible light image and the first original infrared image respectively to determine gradient values, wherein the gradient values ​​include visible light gradient values ​​and infrared gradient values; determining line segment support areas corresponding to the first original visible light image and the first original infrared image according to the gradient values; rectangularizing the line segment support areas, and determining line features in the original visible light image and the infrared image according to the rectangles obtained after the processing.

[0013] In some technical schemes, optionally, determining the registration area based on line features and line features includes: acquiring a historical image database, determining a point feature density threshold and a line feature density threshold based on the historical image database; determining a first point feature density based on at least one first point feature; when the value of the first point feature density is less than the point feature density threshold, determining the area corresponding to the first point feature density as the first registration area; determining a first line feature density based on at least one first line feature; when the value of the first line feature density is less than the line feature density threshold, determining the area corresponding to the first line feature density as the first registration area; determining at least one first registration area corresponding to the original visible light image.

[0014] In some technical schemes, optionally, determining the registration area based on point features and line features also includes: determining a second point feature density based on at least one second point feature; when the value of the second point feature density is less than a point feature density threshold, determining the area corresponding to the second point feature density as the second registration area; determining a second line feature density based on at least one second line feature; when the value of the second line feature density is less than a line feature density threshold, determining the area corresponding to the second line feature density as the second registration area; and determining at least one second registration area corresponding to the original infrared image.

[0015] In some technical schemes, optionally, determining the registration area based on point features and line features also includes: determining the point feature density corresponding to at least one point feature; when the point feature density is greater than the point feature density threshold, determining the area corresponding to at least one point feature as the point-line feature registration area; determining the line feature density corresponding to at least one line feature; when the line feature density is greater than the line feature density threshold, determining the area corresponding to at least one line feature as the point-line feature registration area, the point-line feature registration area includes a first point-line feature registration area and a second point-line feature registration area, the first point-line feature registration area corresponds to the original visible light image, and the second point-line feature registration area corresponds to the original infrared image.

[0016] The second aspect of the present invention provides an image registration device, including: an acquisition module, used to acquire an original visible light image and an original infrared image under the same target scene; a point feature extraction module, used to perform feature extraction on the original visible light image and the original infrared image respectively, and determine the point features; a line feature extraction module, used to perform feature extraction on the original visible light image and the original infrared image respectively, and determine the line features; an area determination module, used to determine the registration area according to the point features and the line features; an initial value calculation module, used to solve the initial value of the registration matrix for the registration area, and determine the initial value of the registration matrix; a registration module, used to establish an optimization equation based on the initial value of the registration matrix combined with the point features and the line features, and determine the registration result of the original visible light image and the original infrared image.

[0017] An embodiment of the third aspect of the present invention provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the steps in the first aspect when executed by the processor.

[0018] An embodiment of the fourth aspect of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the first aspect are implemented.

[0019] An embodiment of the fifth aspect of the present invention provides a chip, the chip includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps in the first aspect.

[0020] The technical solution provided by the present invention brings at least the following beneficial effects:

[0021] The present invention proposes an image registration method, which performs regional registration on infrared images and visible light images acquired from the same scene, performs joint optimization in combination with point feature line feature relationships, outputs a registration matrix and a registration result, and improves the robustness and accuracy of the fusion registration of infrared images and visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 A schematic diagram of a flow chart of an image registration method provided by an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of a partial flow chart of an image registration method provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a partial flow chart of an image registration method provided by an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of a partial flow chart of an image registration method provided by an embodiment of the present invention;

[0028] Figure 5 A schematic diagram of a partial flow chart of an image registration method provided by an embodiment of the present invention;

[0029] Figure 6 A schematic diagram of a partial flow chart of an image registration method provided by an embodiment of the present invention;

[0030] Figure 7 A schematic block diagram of the structure of an image registration device provided by an embodiment of the present invention;

[0031] Figure 8 A schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0032] Fig. 9 A schematic diagram of the process of an image registration method provided by an embodiment of the present invention.

[0033] in, Figure 7 and Figure 8 The corresponding relationship between the component names and numbers in is as follows:

[0034] 900: image registration device; 902: acquisition module; 904: point feature extraction module; 906: line feature extraction module; 908: region determination module; 910: initial value calculation module; 912: registration module; 1000: electronic device; 1109: memory; 1110: processor. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] See also Figure 1 The first aspect of the present invention provides an image registration method, comprising the following steps:

[0037] Step S100: acquiring an original visible light image and an original infrared image of the same target scene;

[0038] Step S102: extracting features from the original visible light image and the original infrared image respectively to determine point features, where the point features include a first point feature and a second point feature, where the first point feature corresponds to the original visible light image and the second point feature corresponds to the original infrared image;

[0039] Step S104: extracting features from the original visible light image and the original infrared image respectively to determine line features, where the line features include a first line feature and a second line feature, where the first line feature corresponds to the original visible light image and the second line feature corresponds to the original infrared image;

[0040] Step S106: determining a registration area according to the point features and the line features, the registration area comprising a first registration area and a second registration area, the first registration area corresponds to the original visible light image, and the second registration area corresponds to the original infrared image;

[0041] Step S108: performing initial value calculation of the registration matrix on the registration area to determine the initial value of the registration matrix;

[0042] Step S110: establishing an optimization equation based on the initial value of the registration matrix combined with the point features and the line features to determine the registration result of the original visible light image and the original infrared image.

[0043] According to the image registration method provided by the present invention, the infrared image and visible light image acquired from the same scene are registered by region, the feature registration region is determined, point features and line features are extracted from the feature registration regions of the infrared image and the visible light image respectively, the regions of the infrared image and the visible light image are divided to determine the registration region, the spatial position relationship between the point features and the line features is used to establish a point-line feature association as a point-line association feature, and finally the result of the regional registration is calculated and used as an initial value, and the point features, line features and point-line association features are used for joint optimization to obtain the registration result of the infrared image and the visible light image. Specifically, the visible light image and the infrared image of the same scene are collected by a visible light sensor and an infrared sensor, and are used as the original visible light image and the original infrared image. Feature extraction is performed on the original visible light image and the original infrared image respectively, the purpose of which is to extract common features, namely, point features and line segment features, from the infrared image and the visible light image, and then establish data association and matching relationship between the infrared image and the visible light image based on the result of feature matching, wherein point feature extraction is performed on the infrared image and the visible light image respectively, feature descriptors corresponding to key points of the infrared image and the visible light image are determined, and at least one point feature corresponding to the infrared image is determined according to the feature descriptor of the infrared image, namely, the second point feature; at least one point feature corresponding to the visible light image is determined according to the feature descriptor of the visible light image, namely, the first point feature. After point feature extraction of the infrared image and the visible light image is completed, line feature extraction is performed on the infrared image and the visible light image respectively, by determining the line segment support area in the image and rectangularizing the line segment support area to obtain a relatively regular rectangular area as the line segment extraction result, and line segments corresponding to rectangular areas in the infrared image and the visible light image are determined respectively, thereby determining at least one line feature corresponding to the infrared image, namely, the second line feature; and at least one line feature corresponding to the visible light image, namely, the first line feature. Feature association is performed based on the extracted point features and line features of the infrared image and the point features and line features of the visible light image. The feature areas of the infrared image and the visible light image are analyzed by setting the point feature density and the line feature density. The high-feature areas corresponding to the infrared image and the visible light image are determined according to the point feature density and the line feature density, that is, the areas where the point feature density and the line feature density are higher than the corresponding thresholds. The data correlation is strong, that is, feature-based registration is performed through point features and line features; and the low-feature areas corresponding to the infrared image and the visible light image are determined according to the point feature density and the line feature density, that is, the areas where the point feature density and the line feature density are lower than the corresponding thresholds. They are registration areas, and the statistical characteristics of the pixel grayscale values ​​are used to determine the similarity between the registration areas corresponding to the visible light image and the infrared image.Specifically, the initial value of the registration matrix of the visible light image and the infrared image is solved through the registration area, and an optimization equation is constructed. After the initial value is calculated, the optimization equation is constructed using the point features of the infrared image, the point features of the visible light image, the line features of the infrared image and the line features of the visible light image. The regional registration problem is converted into a least squares solution problem, and the initial value of the registration matrix is ​​used as the initial value of the least squares optimization iteration to determine the registration result of the infrared image and the visible light image.

[0044] It can be understood that by extracting the point features and line features of the infrared image and the visible light image respectively, and determining the registration area corresponding to the infrared image and the registration area corresponding to the visible light image according to the point features and line features corresponding to each of the infrared image and the visible light image, the region-based registration method and the feature-based registration method are combined, and the result of the regional registration is used as the initial value of the point feature line feature registration, the accuracy of the registration matrix calculation results is improved, thereby improving the fusion accuracy between the visible light image and the infrared image.

[0045] In some embodiments, optionally, Figure 2 As shown, feature extraction is performed on the original visible light image and the original infrared image respectively, and the determined point features include:

[0046] Step S1022: constructing Gaussian pyramids for the original visible light image and the original infrared image respectively, and determining a scale space, where the scale space includes a first scale space and a second scale space, where the first scale space corresponds to the original visible light image, and the second scale space corresponds to the original infrared image;

[0047] Step S1024: determining feature points in the scale space;

[0048] Step S1026: determining a feature descriptor according to the feature points;

[0049] Step S1028: Determine point features in the original visible light image and infrared image according to the feature descriptor.

[0050] In this embodiment, point feature extraction is performed on the acquired original visible light image and original infrared image respectively. Specifically, since the infrared feature texture features in the infrared image are limited, that is, the texture information of the original infrared image is sparse, the scale invariant feature transform (SIFT) method is selected to process the original visible light image and the original infrared image. The corner points of the image, that is, the positions of the points with the local maximum curvature or obvious gradient changes in the image will not be easily changed due to factors such as illumination, radiation transformation and noise. First, a Gaussian pyramid is constructed for the original visible light image and the original infrared image respectively, and each layer is Gaussian blurred using different parameters to construct a Gaussian difference (Difference of Gauss, DOG) scale space, the scale space corresponding to the original visible light image to construct the Gaussian pyramid is the first scale space, and the scale space corresponding to the original infrared image to construct the Gaussian pyramid is the second scale space; determine the feature points corresponding to the original visible light image in the first scale space and the feature points corresponding to the original infrared image in the second scale space, the feature points are local extreme points with directional information detected in images of different scale spaces, these extreme points will not disappear due to changes in lighting conditions, for example: corner points, edge points, bright spots in dark areas and dark spots in bright areas, since there are the same scenes in the two images, then these feature points will have corresponding matches with each other point; a feature descriptor corresponding to the original visible light image and a feature descriptor corresponding to the original infrared image are determined according to the feature points in the first scale space and the feature points in the second scale space, respectively. The feature descriptor is used to determine the point features in the original visible light image and the original infrared image, and is obtained by extracting the feature points in the first scale space and the second scale space. The feature descriptor has the characteristics of discrimination, invariance, robustness and compactness. The feature points are determined by the feature descriptor in such a way that the feature points are easy to distinguish and remain stable, and the feature information around the feature points is described with as few dimensions or information as possible to improve the computational efficiency and storage efficiency.

[0051] In some embodiments, optionally, Figure 3 As shown, feature extraction is performed on the original visible light image and the original infrared image respectively, and the line features are determined to include:

[0052] Step S1042: scaling the original visible light image and the original infrared image respectively to determine a first original visible light image and a first original infrared image;

[0053] Step S1044: performing gradient calculation on the first original visible light image and the first original infrared image respectively to determine gradient values, where the gradient values ​​include visible light gradient values ​​and infrared gradient values;

[0054] Step S1046: determining the line segment support region corresponding to the first original visible light image and the first original infrared image according to the gradient value;

[0055] Step S1048: rectangularizing the line segment support region, and determining line features in the original visible light image and infrared image based on the rectangle obtained after the processing.

[0056] In this embodiment, line features are extracted from the original visible light image and the original infrared image respectively. In this embodiment, a line segment detection algorithm (Line Segment Detector, LSD) is used to detect the straight line segments of the original visible light image and the original infrared image. This method can obtain a high-precision straight line segment detection result in a short time. First, the gradient size and direction of all points in the image are calculated, and then the points with small gradient direction changes and adjacent points are regarded as a connected domain. According to the rectangularity of each domain, it is determined whether it needs to be disconnected according to the rules to form multiple domains with larger rectangularity. Finally, all the generated domains are improved and screened, and the domains that meet the conditions are retained, which are the final line detection results. Specifically, the original visible light image and the original infrared image are scaled respectively, and the scaled images are determined as the first original visible light image and the first original infrared image respectively; the first original visible light image and the first original infrared image are gradient calculated respectively to determine the visible light gradient value corresponding to the first original visible light image and the infrared gradient value corresponding to the first original infrared image, wherein the method for calculating the gradient value includes: performing gradient calculation on the four lower right pixels of each pixel point in the image, assuming that the grayscale of the pixel point (x, y) is i(x, y), then the pixel gradient g of the pixel point on the x-axis and y-axis is x (x, y) and g y (x, y) are:

[0057]

[0058] Among them, x is the coordinate of the pixel point on the x-axis, and y is the coordinate of the pixel point on the y-axis.

[0059] According to the pixel gradient of the pixel point on the x-axis and y-axis, the gradient amplitude G(x, y) and the gradient direction LLA can be obtained:

[0060]

[0061] After determining the pixel gradient, the gradient is sorted. The larger the gradient amplitude calculated by the pixel in the image, the more significant the edge point of the pixel. The threshold of the gradient amplitude is set for screening, and the points with larger amplitude are selected. Then, considering that there are a large number of encouraging pixels around the line segment support domain, an isolated pixel is randomly selected from the sorted list, and the directional tolerance between the gradient direction of the isolated pixel and the gradient direction of the support domain is calculated to see whether it is less than r. If r meets the threshold, the isolated pixel is changed to USED and included in the line segment support domain, and the line segment support domain after adding the isolated pixel is updated. Among them, r is the set directional error tolerance value, which indicates the error value between the rectangular direction of the line segment support domain and the pixel. When r < 22.5, the pixel will be included in the original line segment support area. Then, the updated line segment support domain is rectangularized, and the line segment support domain area is approximated by rectangular calculation to obtain a more regular rectangular area as the line segment extraction result display, and the line segment support area corresponding to the first original visible light image and the line segment support area corresponding to the first original infrared image are determined respectively. Each generated area is judged by setting the pixel density value f in the rectangular area. When the number of pixels in the rectangular line segment support area is greater than the pixel density value f, the corresponding line segment is determined to be a line feature in the first original visible light image or the first original infrared image.

[0062] In some embodiments, optionally, Figure 4 As shown, the registration area is determined based on the line features and the line features, including:

[0063] Step S1062: Acquire a historical image database, and determine a point feature density threshold and a line feature density threshold according to the historical image database;

[0064] Step S1064: determining a first point feature density according to at least one first point feature;

[0065] Step S1066: when the value of the first point feature density is less than the point feature density threshold, determining the area corresponding to the first point feature density as the first registration area;

[0066] Step S1068: determining a first line feature density according to at least one first line feature;

[0067] Step S1070: when the value of the first line feature density is less than the line feature density threshold, determining the area corresponding to the first line feature density as the first registration area;

[0068] Step S1072: Determine at least one first registration region corresponding to the original visible light image.

[0069] In this embodiment, the operator analyzes the images in the historical image database and determines the point feature density threshold k and the line feature density threshold m according to the image point feature density and line feature density in the historical image data, wherein the point feature density threshold k indicates that when the point feature of a certain area in the image is k, the area can be used as a feature area in the image registration process, and the area has strong characteristics; the line feature density m indicates that when the line feature of a certain area in the image is m, the area can be used as a feature area in the image registration process, and the area has strong characteristics. Because the image data in the historical image database in actual applications will retain the historical registration threshold with good registration effect, the setting of m and k can be the average of at least one historical registration threshold. Determine the first point feature corresponding to the original visible light image. When the first point feature density in the original visible light image is less than the point feature density threshold k, determine that the region corresponding to the first point feature density has weak characteristics, that is, the image texture is sparse, and the region can be used for registration by the regional registration method, that is, the first registration region; determine the first line feature corresponding to the original visible light image. When the first line feature density in the original visible light image is less than the point feature density threshold m, determine that the region corresponding to the first line feature density has weak characteristics, that is, the image texture is sparse, and the region can be used for registration by the regional registration method, that is, the first registration region; in the regional registration method, first, take one image as a template and search for the region most similar to the template in another image; secondly, use the statistical characteristics of the pixel gray value to evaluate the similarity between images; finally, adjust the transformation parameters between images so that the similarity index reaches a maximum value, thereby achieving registration.

[0070] In some embodiments, optionally, Figure 5 As shown, the registration area is determined based on point features and line features, and also includes:

[0071] Step S1162: determining a second point feature density according to at least one second point feature;

[0072] Step S1164: when the value of the second point feature density is less than the point feature density threshold, determining the area corresponding to the second point feature density as the second registration area;

[0073] Step S1166: determining a second line feature density according to at least one second line feature;

[0074] Step S1168: when the value of the second line feature density is less than the line feature density threshold, determining the area corresponding to the second line feature density as the second registration area;

[0075] Step S1170: Determine at least one second registration region corresponding to the original infrared image.

[0076] In this embodiment, the second point feature corresponding to the original infrared image is determined. When the second point feature density in the original infrared image is less than the point feature density threshold k, it is determined that the regional characteristic corresponding to the second point feature density is weak, that is, the image texture is sparse, and the region can be used for registration by the regional registration method, that is, the second registration region; the second line feature corresponding to the original infrared image is determined. When the second line feature density in the original infrared image is less than the point feature density threshold m, it is determined that the regional characteristic corresponding to the second line feature density is weak, that is, the image texture is sparse, and the region can be used for registration by the regional registration method, that is, the second registration region.

[0077] In some embodiments, optionally, Figure 6 As shown, the registration area is determined based on point features and line features, and also includes:

[0078] Step S1262: Determine the point feature density corresponding to at least one point feature;

[0079] Step S1264: when the point feature density is greater than the point feature density threshold, determining an area corresponding to at least one point feature as a point-line feature registration area;

[0080] Step S1266: determining a line feature density corresponding to at least one line feature;

[0081] Step S1268: When the line feature density is greater than the line feature density threshold, determine that the area corresponding to at least one line feature is a point-line feature registration area, and the point-line feature registration area includes a first point-line feature registration area and a second point-line feature registration area.

[0082] In this embodiment, when the first point feature density corresponding to the original visible light image is greater than the point feature density threshold k, it is determined that the region corresponding to the first point feature density is more characteristic, and the region can be used for point-line feature registration; when the first line feature density corresponding to the original visible light image is greater than the line feature density threshold m, it is determined that the region corresponding to the first line feature density is more characteristic, and the region can be used for point-line feature registration; the region corresponding to the first point feature density and the first line feature density is the first point-line feature registration region. When the second point feature density corresponding to the original infrared image is greater than the point feature density threshold m, it is determined that the region corresponding to the second point feature density is more characteristic, and the region can be used for point-line feature registration; when the second line feature density corresponding to the original infrared image is greater than the line feature density threshold k, it is determined that the region corresponding to the second line feature density is more characteristic, and the region can be used for point-line feature registration; the region corresponding to the second point feature density and the second line feature density is the second point-line feature registration region. Through the regional registration method combining point features and line features, based on the constraint relationship of feature points or feature lines, the data correlation of the registration of the original visible light image and the original infrared image is improved.

[0083] In a specific embodiment, the image registration method mainly includes point-line association feature extraction, point-line association feature matching, region matching and feature-region joint matching, such as Fig. 9 As shown, the image registration method includes: step S800: acquiring a visible light image; step S900: acquiring an infrared image; step S802: first point feature extraction; step S902: second point feature extraction; step S806: first line feature extraction; step S906: second line feature extraction; step S804: first region division; step S904: second region division; step S808: first region feature extraction; step S908: second region feature extraction; step S810: first point-line feature pairing; step S910: second point-line feature pairing; step S988: initial value of the registration matrix; step S999: joint optimization of multiple features of the visible light image and the infrared image.

[0084] In this embodiment, first, the infrared image and the visible light image are divided into regions to clarify the feature registration region and the regional registration region; secondly, point features and line features are extracted from the feature registration regions of the infrared image and the visible light image, respectively, to perform regional registration of the infrared image and the visible light image; point-line feature associations are established using the spatial positional relationship between point features and line features as point-line association features; finally, the result of regional registration is calculated and used as an initial value, and point features, line features and point-priority association features are used for joint optimization to obtain the registration result of the infrared image and the visible light image.

[0085] Specifically, for point feature extraction, considering that infrared feature texture features are limited, the present invention selects SIFT (Scale Invariant Feature Transform, SIFT), and the corner points will not be easily changed by factors such as illumination, radiation transformation and noise. First, in order to simulate the multi-scale features of image data, a Gaussian pyramid needs to be constructed, and each layer is subjected to Gaussian blur processing with different parameters, thereby constructing a Difference of Gaussian (DOG) scale space; secondly, in the constructed DOG scale space, extreme points are searched as candidate feature points. If a point has the maximum or minimum value in the 26 fields above the current layer and the upper and lower layers of the DOG scale space, the point is considered to be a feature point of the image at this scale; then the gradient direction histogram of each key point is calculated, and a unique vector is claimed, and these vectors are combined to form a feature descriptor of the key point. At this point, the extraction of feature points is completed.

[0086] Line feature extraction, the line feature used in this invention is LSD (Line Segment Detector, LSD), this method can obtain high-precision line detection results in a short time. First, the image is scaled and the LSD gradient is calculated, that is, the gradient of the four lower right pixels of each pixel in the image is calculated. Assuming that the grayscale of the pixel point (x, y) is i(x, y), then the pixel gradient g of the pixel point on the x-axis and y-axis is x (x, y) and g y (x, y) are:

[0087]

[0088] Among them, x is the coordinate of the pixel point on the x-axis, and y is the coordinate of the pixel point on the y-axis.

[0089] According to the pixel gradient of the pixel point on the x-axis and y-axis, the gradient amplitude G(x, y) and the gradient direction LLA can be obtained:

[0090]

[0091] The gradients are sorted. The larger the gradient amplitude calculated by the pixel in the image, the more significant the edge point of the pixel is, and the more suitable it is as the seed point for line segment detection. The threshold of the gradient amplitude is set for screening, and the points with larger amplitude are selected. Then, considering that there are a large number of encouraging pixels around the line segment support domain, an isolated pixel is randomly selected from the sorted list, and the directional tolerance between the gradient direction of the isolated pixel and the gradient direction of the support domain is calculated to see whether it is less than r. If r meets the threshold, the isolated pixel is changed to the used state and included in the line segment support domain, and the line segment support domain after adding the isolated pixel is updated. Among them, r is the set directional error tolerance value, which indicates the error value between the rectangular direction of the line segment support domain and the pixel. When r < 22.5, the pixel will be included in the original line segment support area. Then, the updated line segment support domain is rectangularized, and the line segment support domain area is approximated by a rectangular calculation to obtain a more regular rectangular area as the line segment extraction result.

[0092]

[0093] Among them, τ represents the LLA of the area around the pixel to obtain the gradient angle of all points, l x is the x-axis coordinate of the center of the rectangle, l y is the y-axis coordinate of the center of the rectangle, G(j) is the gradient amplitude of pixel j, ∑ j∈Rejion G(j) represents all the pixels in the support domain of the convenient line segment, and (x(j), y(j)) represents the coordinates of the points in the region. The main direction M of the rectangular region after the line segment support domain is rectangularized is:

[0094]

[0095] in,

[0096]

[0097] Finally, the estimated number of pixels in the rectangle that meet the gradient amplitude can be used to determine whether the rectangle can be used as a "line segment". The more pixels that meet the pixel gradient amplitude, the more likely the rectangle is to be a "line segment". Each generated rectangle is judged by setting the pixel density value f in the rectangle.

[0098] Image region division: divide the image according to the point features and line features calculated above, set appropriate thresholds k and m, and when the point feature density in the infrared image and visible light image is less than k or the line feature density is less than m, perform regional registration on the area, otherwise perform point and line feature registration.

[0099] Multi-feature region joint registration optimization, the initial value of the registration matrix is ​​solved for the region of regional registration, and the optimization equation is constructed:

[0100]

[0101] Where T 0 is the initial value of the registration matrix, I(x, y) is the gray value of the pixel at coordinate (x, y) in the infrared image, and V(x, y) is the gray value of the pixel at coordinate (x, y) in the visible light image. The regional registration problem is transformed into a least squares problem, so as to solve the initial value of the registration matrix. After calculating the initial value, the following optimization equation is constructed using point features and line features:

[0102]

[0103] Where T iv is the registration matrix, where P I (z) is the coordinate of the z feature point in the infrared image, P v (z) is the coordinate of the feature point z in the visible light image, L I (w) is the coordinate of the w line feature in the infrared image, L V (w) is the coordinate of the w line sign in the visible light image. Similarly, the regional registration problem is transformed into a least squares problem. 0 As the initial value of the least squares optimization iteration.

[0104] like Figure 7As shown, the second aspect of the present invention provides an image registration device 900, including: an acquisition module 902, used to acquire an original visible light image and an original infrared image under the same target scene; a point feature extraction module 904, used to perform feature extraction on the original visible light image and the original infrared image respectively, and determine the point features; a line feature extraction module 906, used to perform feature extraction on the original visible light image and the original infrared image respectively, and determine the line features; a region determination module 908, used to determine the registration region according to the point features and the line features; an initial value calculation module 910, used to perform initial value solution of the registration matrix for the registration region, and determine the initial value of the registration matrix; a registration module 912, used to establish an optimization equation based on the initial value of the registration matrix combined with the point features and the line features, and determine the registration result of the original visible light image and the original infrared image.

[0105] like Figure 8 As shown, the third aspect of the present invention provides an electronic device 1000, including a processor 1110, a memory 1109, and a program or instruction stored in the memory 1109 and executable on the processor 1110. When the program or instruction is executed by the processor 1110, the various processes of the above-mentioned image registration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0106] Among them, the processor 1110 is used to obtain the original visible light image and the original infrared image of the same target scene; perform feature extraction on the original visible light image and the original infrared image respectively to determine the point features; perform feature extraction on the original visible light image and the original infrared image respectively to determine the line features; determine the registration area according to the point features and the line features; solve the initial value of the registration matrix for the registration area to determine the initial value of the registration matrix; establish an optimization equation based on the initial value of the registration matrix combined with the point features and the line features to determine the registration result of the original visible light image and the original infrared image.

[0107] Optionally, the processor 1110 is further used to construct a Gaussian pyramid for the original visible light image and the original infrared image respectively, determine the scale space; determine the feature points in the scale space; determine the feature descriptor according to the feature points; and determine the point features in the original visible light image and the infrared image according to the feature descriptor.

[0108] Optionally, the processor 1110 is further used to scale the original visible light image and the original infrared image, respectively, to determine a first original visible light image and a first original infrared image; perform gradient calculations on the first original visible light image and the first original infrared image, respectively, to determine gradient values, the gradient values ​​including visible light gradient values ​​and infrared gradient values; determine line segment support areas corresponding to the first original visible light image and the first original infrared image according to the gradient values; perform rectangular processing on the line segment support areas, and determine line features in the original visible light image and the infrared image according to the rectangles obtained after the processing.

[0109] Optionally, processor 1110 is further used to obtain a historical image database, determine a point feature density threshold and a line feature density threshold based on the historical image database; determine a first point feature density based on at least one first point feature; when the value of the first point feature density is less than the point feature density threshold, determine the area corresponding to the first point feature density as a first registration area; determine a first line feature density based on at least one first line feature; when the value of the first line feature density is less than the line feature density threshold, determine the area corresponding to the first line feature density as a first registration area; determine at least one first registration area corresponding to the original visible light image.

[0110] Optionally, processor 1110 is further used to determine a second point feature density based on at least one second point feature; when the value of the second point feature density is less than a point feature density threshold, determine the area corresponding to the second point feature density as a second registration area; determine a second line feature density based on at least one second line feature; when the value of the second line feature density is less than a line feature density threshold, determine the area corresponding to the second line feature density as a second registration area; determine at least one second registration area corresponding to the original infrared image.

[0111] Optionally, processor 1110 is further used to determine a point feature density corresponding to at least one point feature; when the point feature density is greater than a point feature density threshold, determine the area corresponding to at least one point feature as a point-line feature registration area; determine a line feature density corresponding to at least one line feature; when the line feature density is greater than a line feature density threshold, determine the area corresponding to at least one line feature as a point-line feature registration area.

[0112] In a fourth aspect, the present invention provides a readable storage medium, on which a program or instruction is stored, which, when executed by a processor, implements each process of the above-mentioned image registration method embodiment and can achieve the same technical effect. To avoid repetition, it is not repeated here. In addition, the data storage capacity and data processing speed corresponding to the image registration method in this application are improved by a readable storage medium.

[0113] The methods may be implemented in a variety of different ways depending on the specific features and / or example applications. For example, the methods may be implemented by a combination of hardware, firmware, and / or software. For example, in a hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other device units for performing the above functions, and / or combinations thereof.

[0114] A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices, but is not limited thereto. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory card, floppy disk, encoding mechanical device (such as a punch card or a groove with a raised structure with instructions recorded) and any suitable combination of the above devices. The computer readable storage medium used herein should not be understood as a transmission signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium, or an electrical signal transmitted through a wire, etc.

[0115] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0116] In a fifth aspect, the present invention provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, the processor is used to run a program or instruction, implement each process of the above-mentioned image registration method embodiment, and can achieve the same technical effect, to avoid repetition, no further description is given here. In addition, the chip is used to improve the data processing speed corresponding to the image registration method in this application.

[0117] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the features may work as above in certain combinations and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0118] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0119] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0120] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0121] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An image registration method, characterized in that: include: Obtain the original visible light image and original infrared image of the same target scene; Performing feature extraction on the original visible light image and the original infrared image respectively to determine point features, wherein the point features include a first point feature and a second point feature, wherein the first point feature corresponds to the original visible light image, and the second point feature corresponds to the original infrared image; Extracting features from the original visible light image and the original infrared image respectively to determine line features, wherein the line features include a first line feature and a second line feature, wherein the first line feature corresponds to the original visible light image, and the second line feature corresponds to the original infrared image; Determine a registration area according to the point features and the line features, the registration area comprising a first registration area and a second registration area, the first registration area corresponds to the original visible light image, and the second registration area corresponds to the original infrared image; Performing initial value calculation of the registration matrix on the registration area to determine the initial value of the registration matrix; An optimization equation is established according to the initial value of the registration matrix in combination with the point features and the line features to determine the registration result of the original visible light image and the original infrared image.

2. The image registration method according to claim 1, characterized in that: The extracting features from the original visible light image and the original infrared image respectively to determine point features includes: Constructing Gaussian pyramids for the original visible light image and the original infrared image respectively, and determining a scale space, wherein the scale space includes a first scale space and a second scale space, the first scale space corresponds to the original visible light image, and the second scale space corresponds to the original infrared image; Determining feature points in the scale space; Determine a feature descriptor according to the feature points; Point features in the original visible light image and the infrared image are determined according to the feature descriptor.

3. The image registration method according to claim 1, characterized in that: The extracting features from the original visible light image and the original infrared image respectively to determine line features includes: Respectively scaling the original visible light image and the original infrared image to determine a first original visible light image and a first original infrared image; Performing gradient calculation on the first original visible light image and the first original infrared image respectively to determine gradient values, where the gradient values ​​include visible light gradient values ​​and infrared gradient values; Determine, according to the gradient value, a line segment support region corresponding to the first original visible light image and the first original infrared image; The line segment support region is rectangularized, and line features in the original visible light image and the infrared image are determined according to the rectangles obtained after the processing.

4. The image registration method according to claim 1, characterized in that: The determining of the registration area according to the point features and the line features comprises: Acquire a historical image database, and determine a point feature density threshold and a line feature density threshold according to the historical image database; determining a first point feature density based on at least one of the first point features; When the value of the first point feature density is less than the point feature density threshold, determining the area corresponding to the first point feature density as a first registration area; determining a first line feature density based on at least one of the first line features; When the value of the first line feature density is less than the line feature density threshold, determining the area corresponding to the first line feature density as a first registration area; At least one first registration region corresponding to the original visible light image is determined.

5. The image registration method according to claim 4, characterized in that: The determining of the registration area according to the point features and the line features further includes: determining a second point feature density based on at least one of the second point features; When the value of the second point feature density is less than the point feature density threshold, determining the area corresponding to the second point feature density as the second registration area; determining a second line feature density based on at least one of the second line features; When the value of the second line feature density is less than the line feature density threshold, determining the area corresponding to the second line feature density as a second registration area; At least one second registration region corresponding to the original infrared image is determined.

6. The image registration method according to claim 5, characterized in that: The determining of the registration area according to the point features and the line features further includes: Determine a point feature density corresponding to at least one of the point features; When the point feature density is greater than the point feature density threshold, determining a region corresponding to at least one of the point features as a point-line feature registration region; determining a line feature density corresponding to at least one of the line features; When the line feature density is greater than the line feature density threshold, an area corresponding to at least one of the line features is determined as a point-line feature registration area, and the point-line feature registration area includes a first point-line feature registration area and a second point-line feature registration area, the first point-line feature registration area corresponds to the original visible light image, and the second point-line feature registration area corresponds to the original infrared image.

7. An image registration device, characterized in that: include: An acquisition module, used for acquiring an original visible light image and an original infrared image of the same target scene; A point feature extraction module, used to extract features from the original visible light image and the original infrared image respectively to determine point features; A line feature extraction module, used to extract features from the original visible light image and the original infrared image respectively to determine line features; A region determination module, used for determining a registration region according to the point features and the line features; An initial value calculation module, used to solve the initial value of the registration matrix for the registration area to determine the initial value of the registration matrix; The registration module is used to establish an optimization equation according to the initial value of the registration matrix in combination with the point features and the line features to determine the registration result of the original visible light image and the original infrared image.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or an instruction to implement the steps of the method according to any one of claims 1 to 6.

Images