Registration-based infrared image offset correction method and system

Abstract

The invention relates to the technical field of image processing, in particular to an infrared image offset correction method and system based on registration, and the method comprises the steps: carrying out the zooming and cutting of a reference image and an infrared image, respectively selecting a central pixel on the zoomed and cut reference image and infrared image, and constructing a scale space; dividing the scale space into eight layers, judging whether each pixel point of each layer in the scale space is larger than a preset pixel threshold value and is larger than a pixel of an adjacent point of the pixel point, if so, taking the pixel point as a key point, and calculating the main direction of the key point; the coordinates of the key points are rotated to the main direction of the key points, and descriptors of the key points are calculated after interval division; the descriptors of the key points of the reference image and the descriptors of the key points of the infrared image are respectively extracted for mutual matching, and an affine transformation matrix is established after matching point pairs are obtained. According to the method, the targets of lower calculation time consumption and higher accuracy during infrared image offset correction can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to a registration-based infrared image offset correction method and system. Background Art

[0002] With the widespread application of infrared imaging technology, a large number of valuable infrared images have been obtained in fields such as military reconnaissance, environmental monitoring, and medical diagnosis. During the inspection process, the infrared pan-tilt camera will be offset at the same preset inspection point due to the wear of the drive motor due to long-term operation and the error in the accuracy of the structural parts, which will affect the normal inspection work of the infrared pan-tilt camera. Therefore, it is necessary to correct the offset of the infrared pan-tilt to ensure that the camera returns to the preset position and operates normally. Position offset will affect subsequent image analysis and processing, reduce the accuracy of the analysis, and may even lead to wrong judgments. Therefore, it is particularly important to effectively correct infrared images that have been offset from the preset position.

[0003] Existing position offset correction methods mainly include manual detection and template matching. Among them, although the manual detection method can accurately obtain the precise position of the object to be measured, it requires a lot of manpower and time. The template matching method can quickly find areas similar to the original image in the image through a pre-established template, but the accuracy depends on the selection of the template and requires a certain amount of computing resources.

[0004] Therefore, the present invention provides a registration-based infrared image offset correction method and system. Summary of the invention

[0005] By scaling and cropping the reference image and the infrared image, the central pixel is selected on the scaled and cropped reference image and infrared image respectively, and a scale space is constructed based on the central pixel; the scale space is divided into 8 layers, and for each pixel point in each layer of the scale space, it is determined whether the pixel point is greater than a preset pixel threshold and greater than the pixel of the adjacent point of the pixel point. If so, the pixel point is a key point, and the main direction of the key point is calculated; the coordinates of the key point are rotated to the main direction of the key point, and the main direction of the key point is divided into intervals and then the descriptor of the key point is calculated; the descriptor of the key point of the reference image and the descriptor of the key point of the infrared image are respectively extracted, and they are matched with each other, and an affine transformation matrix is ​​established after obtaining the matching point pair, and the affine transformation matrix is ​​used to correct the preset point of the infrared image that is offset. The present invention can achieve the goal of lower calculation time and higher accuracy when correcting the offset of the infrared image.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a registration-based infrared image offset correction method, which is implemented by executing the following steps:

[0007] Step S1: obtaining a reference image and an infrared image, setting a plurality of preset points on the reference image, and scaling and cropping the reference image and the infrared image respectively;

[0008] Step S2: selecting a central pixel on the scaled and cropped reference image and the infrared image respectively, and constructing a scale space based on the central pixel;

[0009] Step S3: Divide the scale space into 8 layers, and for each pixel point in each layer of the scale space, determine whether the pixel point is greater than a preset pixel threshold and greater than the pixel of the adjacent point of the pixel point. If so, the pixel point is a key point, and the main direction of the key point is calculated;

[0010] Step S4: rotating the coordinates of the key point to the main direction of the key point, and dividing the main direction of the key point into intervals to calculate the descriptor of the key point;

[0011] Step S5: extract the descriptors of the key points of the reference image and the key points of the infrared image respectively, match them with each other, obtain the matching point pairs, and then establish an affine transformation matrix, which is used to correct the preset points of the infrared image that are offset.

[0012] As a preferred technical solution of the present invention, the method for scaling and cropping the reference image includes:

[0013] Determine whether the area of ​​the preset point coordinate matrix in the reference image is less than M times the area of ​​the reference image, 0<M<1;

[0014] If yes, the four boundaries of the cropped area are expanded outward by N1 pixels, where N1 is an integer greater than 1;

[0015] If not, then based on the preset point coordinates, determine whether the preset point coordinate frame is vertical or horizontal; if it is vertical, then expand the preset point coordinate frame horizontally by N2 pixels, where N2 is an integer greater than 1; if it is horizontal, then expand the preset point coordinate frame vertically by N3 pixels, where N3 is an integer greater than 1; and crop the reference image based on the values ​​of N2 and N3.

[0016] As a preferred technical solution of the present invention, the method for scaling and cropping the infrared image includes:

[0017] When the width pixel of the infrared image is less than a first threshold, the infrared image remains unchanged;

[0018] When the width pixel of the infrared image is greater than or equal to the first threshold and less than the second threshold, the infrared image is reduced to half the size of the original image;

[0019] When the width pixel of the infrared image is greater than or equal to the second threshold and less than the third threshold, the infrared image is reduced to one quarter of the size of the original image;

[0020] When the width pixel of the infrared image is greater than or equal to the third threshold, the infrared image is reduced to one eighth of the original image size;

[0021] The cropping area of ​​the infrared image needs to be larger than the cropping area of ​​the reference image, and an offset value N4 is set to expand the boundary of the cropping area of ​​the reference image by N4 pixels, where N4 is an integer greater than 1.

[0022] As a preferred technical solution of the present invention, the method for calculating the main direction of the key point includes:

[0023] Using histograms to count the gradients and directions in the infrared image and the reference image respectively;

[0024] The angle of the gradient histogram is divided into 36 columns, each column is 10°;

[0025] The peak direction of the gradient histogram is taken as the main direction of the key point.

[0026] As a preferred technical solution of the present invention, the method for calculating the descriptor of the key point after the main direction of the key point is divided into intervals includes:

[0027] The main direction of the key point is divided into 8 direction intervals according to the 0°360° gradient direction, and the descriptor of each direction of the key point is calculated respectively.

[0028] As a preferred technical solution of the present invention, the method of extracting the descriptors of the key points of the reference image and the key points of the infrared image respectively and matching them with each other to obtain the matching point pair includes:

[0029] The Euclidean distance between the descriptor of the key point of the reference image and the descriptor of the key point of the infrared image is calculated by formula 1;

[0030]

[0031] Wherein, d represents the Euclidean distance between the corresponding key point descriptors in the reference image and the infrared image, n represents the dimension of the scale space, i represents the i-th dimension in the scale space, and X i1 and X i2 Respectively represent two points of the i-th dimension in the scale space;

[0032] The point pairs satisfying Formula 2 are taken as correct matching point pairs, and the point pairs not satisfying Formula 2 are taken as wrong point pairs;

[0033]

[0034] In which, let Threshold be 1, R i represents the key points in the reference image or the infrared image, S j represents the distance R in the reference image or the infrared image i The nearest point, S p represents the distance R in the reference image or the infrared image i The next closest point.

[0035] As a preferred technical solution of the present invention, the method of establishing the affine transformation matrix includes:

[0036] Calculate the transformation relationship between the reference image and the infrared image:

[0037]

[0038] Coordinates after affine transformation:

[0039]

[0040] Wherein, h represents the affine transformation matrix, (x, y) represents the coordinates before the affine transformation, that is, the preset point coordinates, and w represents the weight of the affine transformation. represents the linear transformation matrix, that is, the sine and cosine values ​​of the x and y affine transformations, [h 02 h 12 ] T Indicates image translation, that is, x and y translation values, respectively, [h 20 h 21 h 22 ]=[0 0 1], (x', y') represents the corrected coordinates of the preset point.

[0041] As a preferred technical solution of the present invention, the method for establishing the affine transformation matrix also includes:

[0042] If the number of the obtained optimal consistent sets is 0, it is determined that there is a problem with the infrared image, and the coordinates of the preset point are returned;

[0043] In the affine transformation matrix, if h 00 and h 11 They all fluctuate around 1.0, and h 01 and h 12If both are less than 100, it indicates that the affine transformation matrix is ​​correct; otherwise, it is determined that the affine transformation matrix is ​​incorrect and there is a problem with the infrared image, and the preset point coordinates are returned.

[0044] As a preferred technical solution of the present invention, a method for constructing a scale space includes:

[0045] The scale space has 8 layers by default, and the construction method of each layer adopts the following process:

[0046] The feature point detection of the sar-sift algorithm proposes a new calculation of gradient amplitude and gradient direction based on SIFT, that is, the neighborhood of the central pixel is divided into four directions: up, down, left, and right, and four different templates are used to convolve the image to obtain the convolution values ​​m of the four parts of the image. up , m down ,m right ,m left , then the horizontal and vertical gradients are expressed as:

[0047]

[0048] Where I represents the image, Gx represents the partial derivative of image I with respect to x, x represents the horizontal direction, Gy represents the partial derivative of image I with respect to y, y represents the vertical direction;

[0049] The expressions for the gradient direction and magnitude are:

[0050]

[0051] The scale space:

[0052]

[0053] R(x,y,σ)=det(C SH (x,y,σ))+t×tr(C SH (x,y,σ) 2 (Formula 8)

[0054] Among them, θ(x,y) represents the gradient direction, m(x,y) represents the amplitude, and C SH (x, y, σ) represents a matrix, and R(x, y, σ) in Formula 7 represents a function. represents a Gaussian kernel, det(*) in Formula 8 represents the value of the matrix determinant, tr(*) represents the trace of the matrix, and t represents the matrix coefficient.

[0055] The present invention also provides an infrared image offset correction system based on registration as described above, comprising the following modules:

[0056] A preprocessing unit, used for acquiring a reference image and an infrared image, setting a plurality of preset points on the reference image, and scaling and cropping the reference image and the infrared image respectively;

[0057] A construction unit, used for selecting a central pixel on the scaled and cropped reference image and the infrared image, respectively, and constructing a scale space based on the central pixel;

[0058] A detection unit, used to divide the scale space into 8 layers, and for each pixel point in each layer of the scale space, determine whether the pixel point is greater than a preset pixel threshold and greater than the pixel of the adjacent point of the pixel point. If so, the pixel point is a key point, and the main direction of the key point is calculated;

[0059] A calculation unit, used for rotating the coordinates of the key point to the main direction of the key point, and calculating the descriptor of the key point after dividing the main direction of the key point into intervals;

[0060] The matching unit is used to extract the descriptors of the key points of the reference image and the key points of the infrared image respectively, and match them with each other. After obtaining the matching point pairs, an affine transformation matrix is ​​established. The affine transformation matrix is ​​used to correct the preset points of the infrared image that are offset.

[0061] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0062] The technical solution of the present invention corrects the offset infrared image through a registration-based method, pre-processes the image considering the time consumption of the algorithm, and adds a fault-tolerant design in the descriptor matching process to ensure the accuracy of the detection, and can automatically make a judgment when a problematic image is input, avoiding crashes and result errors during use. Therefore, the advantages of the present invention are that it achieves lower calculation time consumption, higher correction accuracy, and supports real-time detection and adjustment, and has broad application prospects, especially suitable for technical fields such as intelligent monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0064] Figure 1 is a flowchart of the steps of the infrared image offset correction method based on registration in the present invention;

[0065] Figure 2The reference image in the present invention is an image with correct preset points;

[0066] Figure 3 The infrared image in the present invention is an image with an error in the preset point;

[0067] Figure 4 It is a structural diagram of the infrared image offset correction system based on registration in the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0069] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0070] Existing position offset correction methods mainly include manual detection and template matching. Although the manual detection method can obtain the precise position of the object to be measured more accurately, it requires a lot of manpower and time; while the template matching method can quickly find areas similar to the original image in the image through pre-established templates, but the accuracy depends on the selection of the template and requires a certain amount of computing resources.

[0071] In view of the above technical problems, the present invention proposes Figure 1 The infrared image offset correction method based on registration shown in the figure comprises the following steps:

[0072] Step S1: Acquire a reference image and an infrared image, set a plurality of preset points on the reference image, and scale and crop the reference image and the infrared image respectively.

[0073] Specifically, when acquiring infrared images, the size may be inconsistent due to factors such as equipment and distance, which may affect the accuracy of subsequent processing. Therefore, through the above technical solution, the image size is first adjusted to match the reference image, and then the image is cropped by setting preset points to remove redundant information at the edge of the image and reduce the amount of calculation for subsequent processing. Figure 2 The reference image shown is the image with the correct preset points; Figure 3 The infrared image shown is an image with an incorrect preset point. There is a gray box in the red line in the image, indicating an incorrect preset point.

[0074] Step S2: Select central pixels on the scaled and cropped reference image and infrared image respectively, and construct a scale space based on the central pixels.

[0075] Specifically, the target features in the infrared image may not be obvious and difficult to be used directly for matching. Based on the different features of infrared images at different scales, the above method is used to process scale changes, and the features in the image are enhanced through convolution and gradient calculation to make them more prominent. By constructing a scale space, key points can be detected at different scales to improve the accuracy of the algorithm. And by obtaining the horizontal and vertical gradients of the infrared image, as well as the gradient direction and gradient amplitude, directional information is provided for the key points, laying the foundation for subsequent processing.

[0076] Step S3: Divide the scale space into 8 layers, and for each pixel point in each layer of the scale space, determine whether the pixel point is greater than a preset pixel threshold and greater than the pixels of the pixel's adjacent points. If so, the pixel point is a key point, and the main direction of the key point is calculated.

[0077] Specifically, since the key points have different directions in different images, in order to avoid affecting the matching effect, in the above scheme, the candidate key points are determined by comparing adjacent points, and the real key points are screened out to improve the detection accuracy. At the same time, the main direction of the key points is calculated so that the key points have rotation invariance, thereby improving the matching stability.

[0078] Step S4: Rotate the coordinates of the key point to the main direction of the key point, divide the main direction of the key point into intervals, and calculate the descriptor of the key point.

[0079] Specifically, traditional feature descriptors may not be able to fully describe the features of key points, while high-dimensional feature descriptors will increase computational complexity and reduce matching efficiency. Therefore, the above technical solution adopts the GLOH descriptor to construct a rich feature descriptor through 17 location bins and 8 orientation intervals to fully describe the features of key points, and while ensuring the adequacy of the description, it reduces the dimension of the features and improves the matching efficiency through reasonable bin and interval division. Among them, GLOH (Gradient Location and Orientation Histogram) is a method for image feature description, which expands the traditional SIFT (Scale Invariant Feature Transform) descriptor and improves the robustness and uniqueness of feature description by introducing more complex grid division and calculation of gradient orientation histogram.

[0080] Step S5: extract the descriptors of the key points of the reference image and the key points of the infrared image respectively, match them with each other, and establish an affine transformation matrix after obtaining the matching point pairs. The affine transformation matrix is ​​used to correct the preset points of the infrared image that are offset.

[0081] Specifically, there may be mismatched point pairs in the matching process. In order not to affect the accuracy of the transformation matrix, the most similar matching point pairs are obtained by matching key points with descriptors, and matching point pairs that do not meet the model are eliminated to improve the accuracy of the matching. The affine transformation matrix is ​​estimated based on the correct matching point pairs to correct the preset points that are offset in the infrared image and achieve accurate image registration. Among them, the matching point pairs that do not meet the model refer to the point pairs that do not meet the following formula 2, that is, the wrong point pairs. Through the above steps, the preset points that are offset in the infrared image are finally corrected successfully.

[0082] Through the coordination between the above steps, compared with the current preset position offset correction method, the present invention is less time-consuming, can automatically correct the offset infrared image, ensure the accuracy of the equipment operation, and can also be used in the field of intelligent monitoring technology.

[0083] Furthermore, the method of scaling and cropping the reference image includes:

[0084] Determine whether the area of ​​the preset point coordinate matrix in the reference image is less than M times the area of ​​the reference image, 0<M<1;

[0085] If yes, the four boundaries of the cropped area are expanded outward by N1 pixels, where N1 is an integer greater than 1;

[0086] If not, based on the preset point coordinates, determine whether the preset point coordinate frame is vertical or horizontal. If it is vertical, expand the preset point coordinate frame horizontally by N2 pixels, where N2 is an integer greater than 1; if it is horizontal, expand the preset point coordinate frame vertically by N3 pixels, where N3 is an integer greater than 1; and crop the reference image based on the values ​​of N2 and N3.

[0087] Specifically, when the correct image of the preset point is cropped through the above steps, in order to avoid the preset point coordinates being on the boundary of the cropping area, the four boundaries of the cropping area are expanded outward by 10 pixels to prevent errors in the result.

[0088] Furthermore, the method for scaling and cropping the infrared image includes:

[0089] When the width pixel of the infrared image is smaller than the first threshold, the infrared image remains unchanged;

[0090] When the width pixel of the infrared image is greater than or equal to the first threshold and less than the second threshold, the infrared image is reduced to half the size of the original image;

[0091] When the width pixel of the infrared image is greater than or equal to the second threshold and less than the third threshold, the infrared image is reduced to one quarter of the original image size;

[0092] When the width pixel of the infrared image is greater than or equal to the third threshold, the infrared image is reduced to one eighth of the original image size;

[0093] The cropping area of ​​the infrared image needs to be larger than the cropping area of ​​the reference image. An offset value N4 is set to expand the border of the cropping area of ​​the reference image by N4 pixels, where N4 is an integer greater than 1.

[0094] Specifically, in order to obtain more key points and ensure the correct matching of the images, the cropping area of ​​the preset point error image needs to be larger than the cropping area of ​​the preset point correct image, that is, a preset offset value offset is set, and the boundary area is expanded based on the cropping area of ​​the preset point correct image to add an offset.

[0095] Furthermore, the method for calculating the main direction of the key point includes:

[0096] Histograms are used to count the gradients and directions in the infrared image and the reference image respectively;

[0097] The angle of the gradient histogram is divided into 36 columns, each column is 10°;

[0098] The peak direction of the gradient histogram is taken as the main direction of the key point.

[0099] Furthermore, the method of calculating the descriptor of the key point after the main direction of the key point is divided into intervals includes:

[0100] The main direction of the key point is divided into 8 direction intervals according to the 0°360° gradient direction, and the descriptors of each direction of the key point are calculated separately.

[0101] Furthermore, the descriptors of the key points of the reference image and the descriptors of the key points of the infrared image are extracted respectively, and the descriptors are matched with each other. The method for obtaining the matching point pair includes:

[0102] The Euclidean distance between the descriptor of the key point of the reference image and the descriptor of the key point of the infrared image is calculated by formula 1;

[0103]

[0104] Where d represents the Euclidean distance between the corresponding key point descriptors in the reference image and the infrared image, n represents the dimension of the scale space, i represents the i-th dimension in the scale space, and X i1 and X i2 Respectively represent two points in the i-th dimension of the scale space;

[0105] The point pairs that satisfy Formula 2 are taken as the correct matching point pairs, and the point pairs that do not satisfy Formula 2 are taken as the wrong point pairs;

[0106]

[0107] In which, let Threshold be 1, R i represents the key points in the reference image or infrared image, S j Represents the distance R in the reference image or infrared image i The nearest point, S p Represents the distance R in the reference image or infrared image i The next closest point.

[0108] Furthermore, the method of establishing an affine transformation matrix includes:

[0109] Calculate the transformation relationship between the reference image and the infrared image:

[0110]

[0111] Coordinates after affine transformation:

[0112]

[0113] Among them, h represents the affine transformation matrix, (x, y) represents the coordinates before the affine transformation, that is, the preset point coordinates, and w represents the weight of the affine transformation. represents the linear transformation matrix, that is, the sine and cosine values ​​of the x and y affine transformations, [h 02 h 12 ] T Indicates image translation, that is, x and y translation values, respectively, [h 20 h 21 h 22 ]=[0 0 1], (x', y') represents the corrected coordinates of the preset point.

[0114] Specifically, through the above steps, the affine transformation matrix can be obtained to achieve accurate image registration. However, when the offset between the input reference image and the image to be registered is too large, there is no identical object in the two images, or the two input images are completely unrelated, this algorithm cannot match feature points and calculate the correct affine transformation matrix, or the corrected preset point coordinates are wrong. At this time, in order to avoid program crashes or errors in preset point correction results due to problems with the input image, this algorithm also provides the following technical solutions to solve this problem. Among them, feature points are key points on infrared images.

[0115] Furthermore, the method of establishing an affine transformation matrix also includes:

[0116] If the number of the best consistent sets obtained is 0, it is determined that there is a problem with the infrared image and the coordinates of the preset points are returned;

[0117] In the affine transformation matrix, if h 00 and h 11They all fluctuate around 1.0, and h 01 and h 12 If both are less than 100, it means that the affine transformation matrix is ​​correct. Otherwise, it is determined that the affine transformation matrix is ​​incorrect and there is a problem with the infrared image, and the preset point coordinates are returned.

[0118] Furthermore, the method of constructing the scale space includes:

[0119] The scale space has 8 layers by default, and each layer is constructed using the following process:

[0120] The feature point detection of the sar-sift algorithm proposes a new calculation of gradient amplitude and gradient direction based on SIFT, that is, the neighborhood of the central pixel is divided into four directions: up, down, left, and right, and four different templates are used to convolve the image to obtain the convolution value m of the four parts of the image. up , m down ,m right ,m left , then the horizontal and vertical gradients are expressed as:

[0121]

[0122] Among them, I represents the image, Gx represents the partial derivative of image I with respect to x, x represents the horizontal direction, Gy represents the partial derivative of image I with respect to y, and y represents the vertical direction.

[0123] The expressions for the gradient direction and magnitude are:

[0124]

[0125] Scale space:

[0126]

[0127] R(x,y,σ)=det(C SH (x,y,σ))+t×tr(C SH (x,y,σ) 2 (Formula 8)

[0128] Among them, θ(x,y) represents the gradient direction, m(x,y) represents the amplitude, and C SH (x, y, σ) represents a matrix, and R(x, y, σ) in Formula 7 represents a function. represents the Gaussian kernel, det(*) in Formula 8 represents the value of the matrix determinant, tr(*) represents the trace of the matrix, and t represents the matrix coefficient.

[0129] Specifically, C SH(x, y, σ) represents a matrix, which can be a sar-harris matrix, and R(x, y, σ) represents a function, which can be a sar-harris function.

[0130] Furthermore, in the above step S3, determining the key point from the plurality of candidate key points further comprises:

[0131] The position and scale of the candidate key points are determined by fitting the Taylor series expansion, and whether the contrast of the candidate key points is greater than the preset threshold is checked. The candidate key points and edge response candidate key points whose contrast is less than the preset threshold are removed, and the remaining candidate key points are called key points.

[0132] Specifically, after determining the exact position and scale, the next step is to check whether the contrast of the key point is high enough. The contrast can be estimated by the quadratic term in the Taylor series expansion, that is, calculating the determinant value of the Hessian matrix. If the absolute value of the determinant is less than a preset threshold, the contrast of the key point is considered to be too low and should be removed. In addition to the contrast check, it is also necessary to check whether the key point is located on the edge. Points on the edge are usually unstable because they are very sensitive to noise and image changes. This can be achieved by checking the ratio of the trace of the Hessian matrix (i.e., the sum of the diagonal elements) to the determinant value. If the ratio is too high, it indicates that the point may be an edge response and should be removed. After the above two steps of screening, the remaining candidate key points are considered to be high-quality key points and can be used in steps such as feature description and matching.

[0133] The present invention also provides a Figure 2 The infrared image offset correction system based on registration shown includes the following modules:

[0134] The preprocessing unit is used to obtain a reference image and an infrared image, set a plurality of preset points on the reference image, and respectively scale and crop the reference image and the infrared image.

[0135] The construction unit is used to select a central pixel on the scaled and cropped reference image and infrared image respectively, and construct a scale space based on the central pixel.

[0136] The detection unit is used to divide the scale space into 8 layers, and for each pixel point in each layer of the scale space, determine whether the pixel point is greater than a preset pixel threshold and greater than the pixels of the adjacent points of the pixel point. If so, the pixel point is a key point, and the main direction of the key point is calculated.

[0137] The calculation unit is used to rotate the coordinates of the key point to the main direction of the key point, and calculate the descriptor of the key point after dividing the main direction of the key point into intervals.

[0138] The matching unit is used to extract the descriptors of the key points of the reference image and the key points of the infrared image respectively, and match them with each other. After obtaining the matching point pairs, an affine transformation matrix is ​​established. The affine transformation matrix is ​​used to correct the preset points that are offset in the infrared image.

[0139] In summary, the present invention corrects the offset infrared image based on the registration method, pre-processes the image considering the time consumption of the algorithm, and adds a fault-tolerant design in the descriptor matching process to ensure the accuracy of the detection, and can automatically make a judgment when a problematic image is input, avoiding crashes and result errors during use. Therefore, the advantages of the present invention are that it achieves lower calculation time consumption, higher correction accuracy, and supports real-time detection and adjustment, and has broad application prospects, especially suitable for technical fields such as intelligent monitoring.

[0140] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0142] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0144] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A registration-based infrared image offset correction method, characterized in that: The method comprises the following steps: Step S1: obtaining a reference image and an infrared image, setting a plurality of preset points on the reference image, and scaling and cropping the reference image and the infrared image respectively; Step S2: selecting a central pixel on the scaled and cropped reference image and the infrared image respectively, and constructing a scale space based on the central pixel; Step S3: Divide the scale space into 8 layers, and for each pixel point in each layer of the scale space, determine whether the pixel point is greater than a preset pixel threshold and greater than the pixel of the adjacent point of the pixel point. If so, the pixel point is a key point, and the main direction of the key point is calculated; Step S4: rotating the coordinates of the key point to the main direction of the key point, and dividing the main direction of the key point into intervals to calculate the descriptor of the key point; Step S5: extract the descriptors of the key points of the reference image and the key points of the infrared image respectively, match them with each other, obtain the matching point pairs, and then establish an affine transformation matrix, which is used to correct the preset points of the infrared image that are offset.

2. The method according to claim 1, characterized in that The method for scaling and cropping the reference image includes: Determine whether the area of ​​the preset point coordinate matrix in the reference image is less than M times the area of ​​the reference image, 0<M<1; If yes, the four boundaries of the cropped area are expanded outward by N1 pixels, where N1 is an integer greater than 1; If not, then based on the preset point coordinates, determine whether the preset point coordinate frame is vertical or horizontal; if it is vertical, then expand the preset point coordinate frame horizontally by N2 pixels, where N2 is an integer greater than 1; if it is horizontal, then expand the preset point coordinate frame vertically by N3 pixels, where N3 is an integer greater than 1; and crop the reference image based on the values ​​of N2 and N3.

3. The method according to claim 2, characterized in that The method for scaling and cropping the infrared image includes: When the width pixel of the infrared image is less than a first threshold, the infrared image remains unchanged; When the width pixel of the infrared image is greater than or equal to the first threshold and less than the second threshold, the infrared image is reduced to half the size of the original image; When the width pixel of the infrared image is greater than or equal to the second threshold and less than the third threshold, the infrared image is reduced to one quarter of the size of the original image; When the width pixel of the infrared image is greater than or equal to the third threshold, the infrared image is reduced to one eighth of the original image size; The cropping area of ​​the infrared image needs to be larger than the cropping area of ​​the reference image, and an offset value N4 is set to expand the boundary of the cropping area of ​​the reference image by N4 pixels, where N4 is an integer greater than 1.

4. The method according to claim 1, characterized in that: The method for calculating the main direction of the key point includes: Using histograms to count the gradients and directions in the infrared image and the reference image respectively; The angle of the gradient histogram is divided into 36 columns, each column is 10°; The peak direction of the gradient histogram is taken as the main direction of the key point.

5. The method according to claim 1, characterized in that: The method for calculating the descriptor of the key point after the main direction of the key point is divided into intervals comprises: The main direction of the key point is divided into 8 direction intervals according to the 0°360° gradient direction, and the descriptor of each direction of the key point is calculated respectively.

6. The method according to claim 1, characterized in that The method of respectively extracting the descriptors of the key points of the reference image and the key points of the infrared image and matching them with each other to obtain the matching point pair includes: The Euclidean distance between the descriptor of the key point of the reference image and the descriptor of the key point of the infrared image is calculated by formula 1; Wherein, d represents the Euclidean distance between the corresponding key point descriptors in the reference image and the infrared image, n represents the dimension of the scale space, i represents the i-th dimension in the scale space, and X i1 and X i2 Respectively represent two points of the i-th dimension in the scale space; The point pairs satisfying Formula 2 are taken as correct matching point pairs, and the point pairs not satisfying Formula 2 are taken as wrong point pairs; In which, let Threshold be 1, R i represents the key points in the reference image or the infrared image, S j represents the distance R in the reference image or the infrared image i The nearest point, S p represents the distance R in the reference image or the infrared image i The next closest point.

7. The method according to claim 6, characterized in that The method of establishing the affine transformation matrix includes: Calculate the transformation relationship between the reference image and the infrared image: Coordinates after affine transformation: Wherein, h represents the affine transformation matrix, (x, y) represents the coordinates before the affine transformation, that is, the preset point coordinates, and w represents the weight of the affine transformation. represents the linear transformation matrix, that is, the sine and cosine values ​​of the x and y affine transformations, [h 02 h 12 ] T Indicates image translation, that is, x and y translation values, respectively, [h 20 h 21 h 22 ]=[001], (x', y') represents the corrected coordinates of the preset point.

8. The method according to claim 7, characterized in that The method of establishing the affine transformation matrix also includes: If the number of the obtained optimal consistent sets is 0, it is determined that there is a problem with the infrared image, and the coordinates of the preset point are returned; In the affine transformation matrix, if h 00 and h 11 They all fluctuate around 1.0, and h 01 and h 12 If both are less than 100, it indicates that the affine transformation matrix is ​​correct; otherwise, it is determined that the affine transformation matrix is ​​incorrect and there is a problem with the infrared image, and the preset point coordinates are returned.

9. The method according to claim 1, characterized in that: Methods for constructing scale space include: The scale space has 8 layers by default, and the construction method of each layer adopts the following process: The feature point detection of the sar-sift algorithm proposes a new calculation of gradient amplitude and gradient direction based on SIFT, that is, the neighborhood of the central pixel is divided into four directions: up, down, left, and right, and four different templates are used to convolve the image to obtain the convolution values ​​m of the four parts of the image. up , m down ,m right ,m left , then the horizontal and vertical gradients are expressed as: Where I represents the image, Gx represents the partial derivative of image I with respect to x, x represents the horizontal direction, Gy represents the partial derivative of image I with respect to y, y represents the vertical direction; The expressions for the gradient direction and magnitude are: The scale space: R(x,y,σ) = det(C SH (x,y,σ)) + t × tr(C SH (x,y,σ)) 2 (Equation 8) Among them, θ(x,y) represents the gradient direction, m(x,y) represents the amplitude, and C SH (x, y, σ) represents a matrix, and R(x, y, σ) in Formula 7 represents a function. represents a Gaussian kernel, det(*) in Formula 8 represents the value of the matrix determinant, tr(*) represents the trace of the matrix, and t represents the matrix coefficient.

10. A registration-based infrared image offset correction system, used to implement the method according to any one of claims 1 to 9, characterized in that: The system includes the following modules: A preprocessing unit, used for acquiring a reference image and an infrared image, setting a plurality of preset points on the reference image, and scaling and cropping the reference image and the infrared image respectively; A construction unit, used for selecting a central pixel on the scaled and cropped reference image and the infrared image, respectively, and constructing a scale space based on the central pixel; A detection unit, used to divide the scale space into 8 layers, and for each pixel point in each layer of the scale space, determine whether the pixel point is greater than a preset pixel threshold and greater than the pixel of the adjacent point of the pixel point. If so, the pixel point is a key point, and the main direction of the key point is calculated; A calculation unit, used for rotating the coordinates of the key point to the main direction of the key point, and calculating the descriptor of the key point after dividing the main direction of the key point into intervals; The matching unit is used to extract the descriptors of the key points of the reference image and the key points of the infrared image respectively, and match them with each other. After obtaining the matching point pairs, an affine transformation matrix is ​​established. The affine transformation matrix is ​​used to correct the preset points of the infrared image that are offset.

Images