Image registration method and image registration device

Abstract

The invention provides an image registration method and an image registration device, which are applied to the technical field of image processing. Comprising the following steps: acquiring a target reference image and a first target to-be-registered image of a to-be-detected area containing FOD; constructing an optimal affine transformation model according to the target reference image and the N preset pitch angles, and further performing linear transformation on the first target to-be-registered image to obtain a second target to-be-registered image; according to a Gaussian difference pyramid of a second target to-be-registered image, screening an extreme point of which the principal curvature does not exceed a preset principal curvature threshold as a first feature point, and further screening a first feature point of which the re-projection error and the sampling probability meet a preset condition as a second feature point; and solving a homogeneous linear equation set of the pixel point pairs matched with the second feature points by adopting an SVD method to obtain a transformation homography matrix, and then registering the first target to-be-registered image to obtain a target image. The contradictory closed loop of low efficiency-insufficient precision is broken through, and the aviation operation safety is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image registration method and an image registration device. BACKGROUND

[0002] The airport runway, the apron and the taxiway waiting for the inspection area are the core areas for the take-off and landing of aircraft (such as airplanes), and foreign objects (Foreign Object Debris, FOD) in the inspection area may cause serious safety accidents. Non-objects belonging to the aircraft itself, such as metal fragments, tools, gravel, plastic products and bird carcasses, may be sucked into the aircraft engine or penetrate the fuselage, causing mechanical failure or even crashing. Since the accident rate of the aircraft is positively correlated with the number of foreign objects, real-time and accurate FOD detection has become a key link to ensure the safety of aviation operation. At present, the multi-technical solution combining millimeter wave radar, optical imaging and artificial intelligence (Artificial Intelligence, AI) algorithm is the mainstream of FOD detection, and image registration as a core link of FOD detection directly determines the detection accuracy and robustness.

[0003] The problem of the traditional image registration method is that speed and accuracy are difficult to achieve, and it is impossible to break the contradiction of "low efficiency- insufficient accuracy", which leads to the inability to efficiently and accurately realize FOD detection, and further seriously restricts the efficiency of airport safety inspection. SUMMARY

[0004] The embodiments of the present application provide an image registration method and an image registration device, which pre-align the image through an optimal affine transformation model, effectively reduce the computational complexity of the image registration process, and reduce the computational load to improve the image registration efficiency, thereby shortening the single-frame processing time from minutes to seconds, meeting the real-time requirements of high-frequency detection of electronic devices. At the same time, the three-dimensional extreme value search of the Gaussian difference pyramid combined with the principal curvature constraint effectively filters the edge noise points, and the improved RANSAC algorithm selects the feature points through double conditions (reprojection error and sampling probability), effectively reduces the mismatch rate under the conditions of large camera angle difference, rain and fog, and dark light, and improves the image registration accuracy, thereby reducing the risk of missing foreign objects. That is, this method can break the contradiction of "low efficiency- insufficient accuracy", and can efficiently and accurately realize FOD detection in the future, thereby improving the efficiency of airport safety inspection and ensuring the safety of aviation operation.

[0005] The embodiments of the present application provide an image registration method, comprising: Obtaining a target reference image and a first target to-be-registered image corresponding to a to-be-inspected area containing foreign objects FOD; According to the target reference image and N preset pitch angles, an optimal affine transformation model is constructed, N is an integer greater than 1, and the first target image to be registered is linearly transformed by using the optimal affine transformation model to obtain a second target image to be registered; A three-dimensional extreme value search is performed on a Gaussian difference pyramid corresponding to the second target image to be registered, and an extreme value point with a principal curvature not exceeding a preset principal curvature threshold is screened as a first feature point of the second target image to be registered; and an improved random sample consensus (RANSAC) algorithm is used on all first feature points, and a feature point with a reprojection error and a sampling probability satisfying a preset condition is screened as a second feature point. A singular value decomposition (SVD) method is used to solve a homogeneous linear equation set of pixel point pairs matched with the second feature point, to obtain a variable homography matrix; and the first target image to be registered is registered according to the variable homography matrix, to obtain a target image.

[0006] According to the image registration method provided in the embodiments of the present application, the optimal affine transformation model is constructed according to the target reference image and N preset pitch angles, including: obtaining the N preset pitch angles from a preset pitch angle range; constructing N affine transformation models according to the N preset pitch angles and respective corresponding optimal azimuth angles; performing feature point detection on the target reference image by using the N affine transformation models respectively, to obtain a feature point quantity and a spatial distribution entropy corresponding to each affine transformation model; determining n affine transformation models from the N affine transformation models in descending order of the feature point quantity, 0 < n < N; and taking an affine transformation model with the maximum spatial distribution entropy in the n affine transformation models as the optimal affine transformation model.

[0007] According to the image registration method provided in the embodiment of the present application, the Gaussian difference pyramid includes M Gaussian difference images, M is an integer greater than 2, the M Gaussian difference images include a target Gaussian difference image, and the target Gaussian difference image is a Gaussian difference image adjacent to two Gaussian difference images; the three-dimensional extreme value search is performed on the Gaussian difference pyramid corresponding to the second target image to be registered, and an extreme value point with a principal curvature not exceeding a preset principal curvature threshold is screened as a first feature point of the second target image to be registered, including: traversing each candidate pixel point in the target Gaussian difference image, and obtaining a first pixel point sharing a corner point with the candidate pixel point; determining a second pixel point corresponding to the positions of the candidate pixel point and the first pixel point from two Gaussian difference images adjacent to the target Gaussian difference image; taking a pixel point with the maximum or minimum brightness value from the candidate pixel point, the first pixel point and the second pixel point as an extreme value point; determining the principal curvature according to the Hessian matrix of the extreme value point; and screening an extreme value point with a principal curvature not exceeding a preset principal curvature threshold from all the extreme value points as a first feature point of the second target image to be registered.

[0008] According to the image registration method provided in the embodiment of the present application, the improved random sample consensus (RANSAC) algorithm is used on all the first feature points, a feature point with a re-projection error and a sampling probability satisfying a preset condition is screened as a second feature point, including: randomly selecting P1 first sub-feature points and P2 second sub-feature points from the first feature points, P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of the first feature points; calculating a homography matrix according to the P1 first sub-feature points; for each second sub-feature point, determining a re-projection error according to the homography matrix, the pixel coordinates of the second sub-feature point and the pixel coordinates of a pixel point corresponding to the second sub-feature point in the second target image to be registered; determining a sampling probability according to the re-projection error; and screening a second sub-feature point with a re-projection error less than a preset error threshold and a sampling probability greater than a preset probability threshold from the P2 second sub-feature points as the second feature point.

[0009] According to the image registration method provided in the embodiment of the present application, the singular value decomposition (SVD) method is used to solve a homogeneous linear equation set of pixel point pairs matched with the second feature points to obtain a transformation homography matrix, including: determining a pixel point matched with each second feature point from the target reference image according to the descriptor of all the second feature points, and constructing a pixel point pair; constructing a homogeneous equation according to the pixel coordinates of each pixel point pair; and solving a homogeneous linear equation set constructed by all the homogeneous equations by using the SVD method to obtain the transformation homography matrix.

[0010] According to the image registration method provided in the embodiment of the present application, the N affine transformation models are constructed according to the N preset tilt angles and the respective optimal azimuth angles, including: for each preset tilt angle, the following operations are performed: a tilt parameter is determined according to the preset tilt angle, the calculation formula of the tilt parameter is: , denotes the tilt parameter, denotes the preset tilt angle; an azimuth angle sampling interval is determined according to the tilt parameter, the calculation formula of the azimuth angle sampling interval is: , denotes the azimuth angle sampling interval, denotes a constant; an optimal azimuth angle is determined according to the azimuth angle sampling interval; and an affine transformation model is constructed according to the optimal azimuth angle and the preset tilt angle.

[0011] According to the image registration method provided in the embodiment of the present application, the principal curvature is determined according to the Hessian matrix of the extreme point, including: the Hessian matrix is constructed according to the first second-order partial derivative of the extreme point in the direction, the second second-order partial derivative of the extreme point in the direction, and the cross partial derivative of the first second-order partial derivative and the second second-order partial derivative; the principal curvature is determined according to the Hessian matrix, and the calculation formula of the principal curvature is: ; wherein, denotes the principal curvature; denotes the Hessian matrix; denotes the summation result of the first second-order partial derivative and the second second-order partial derivative; denotes the difference result between the product result of the first second-order partial derivative and the second second-order partial derivative and the square result of the cross partial derivative.

[0012] According to the image registration method provided in the embodiment of the present application, the homogeneous linear equation set is , denotes a coefficient matrix, denotes a flattened vector constructed by all elements in the variable single homographic matrix, and the SVD method is adopted to solve the homogeneous linear equation set constructed by all homogeneous equations to obtain the variable single homographic matrix, including: the SVD method is adopted to decompose the coefficient matrix to obtain a right singular vector matrix ; the last column element in the right singular vector matrix is taken as the flattened vector ; and the variable single homographic matrix is constructed according to all elements in the flattened vector .

[0013] According to the image registration method provided in the embodiment of the present application, the target reference image and the first target image to be registered corresponding to the region to be inspected containing foreign objects FOD are obtained, which comprises: acquiring images collected by a multi-view camera array on the region to be inspected; taking the image collected by one camera in the multi-view camera array as a first reference image and taking the images collected by the remaining cameras as first images to be registered; performing image scaling on the first reference image and the first image to be registered respectively by a preset sampling operator to obtain a second reference image and a second image to be registered; and performing high-frequency noise elimination on the second reference image and the second image to be registered respectively by a Gaussian filter to obtain the target reference image and the first target image to be registered.

[0014] The embodiment of the present application further provides an image registration device, comprising: The acquisition module is used to acquire a target reference image and a first target image to be registered corresponding to a region to be inspected containing foreign objects FOD; The transformation module is used to construct an optimal affine transformation model according to the target reference image and N preset pitch angles, N is an integer greater than 1, and perform linear transformation on the first target image to be registered by using the optimal affine transformation model to obtain a second target image to be registered; The screening module is used to perform three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image to be registered, screen extreme value points with a principal curvature not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered, and screen feature points with a re-projection error and a sampling probability satisfying a preset condition as second feature points by using an improved random sample consensus RANSAC algorithm on all first feature points; The registration module is used to solve a homogeneous linear equation group of pixel point pairs matched with the second feature points by using a singular value decomposition SVD method to obtain a transformation homography matrix, and perform registration on the first target image to be registered according to the transformation homography matrix to obtain a target image.

[0015] The embodiment of the present application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the image registration method of any of the above when executing the computer program.

[0016] The embodiment of the present application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the image registration method of any of the above.

[0017] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the image registration method of any of the above.

[0018] The image registration method and the image registration device provided by the embodiments of the present application, by obtaining a target reference image and a first target image to be registered corresponding to a to-be-inspected area containing foreign objects FOD; constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, N being an integer greater than 1, and performing linear transformation on the first target image to be registered by using the optimal affine transformation model to obtain a second target image to be registered; performing three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image to be registered, and screening extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered; and screening feature points with a re-projection error and a sampling probability satisfying a preset condition as second feature points by using an improved random sample consensus RANSAC algorithm on all first feature points; solving a homogeneous linear equation set of pixel pairs matched with the second feature points by using a singular value decomposition SVD method to obtain a transformation homography matrix; and performing registration on the first target image to be registered according to the transformation homography matrix to obtain a target image. The method effectively reduces the calculation complexity of the image registration process and the calculation load by pre-aligning the image through the optimal affine transformation model, so as to improve the image registration efficiency, and then shortens the single-frame processing time from minutes to seconds, so as to meet the real-time requirement of high-frequency detection of electronic equipment. At the same time, the three-dimensional extreme value search of the Gaussian difference pyramid combined with the principal curvature constraint effectively filters the edge noise points, and the improved RANSAC algorithm screens the feature points through the double conditions (the re-projection error and the sampling probability), so as to effectively reduce the mismatching rate under the conditions of large camera view angle difference and rain, fog and dark light, and improve the image registration accuracy, and then reduce the risk of foreign object missed detection. That is, the method can break through the contradiction of "low efficiency - insufficient accuracy", and can efficiently and accurately realize FOD detection in the future, so as to improve the airport safety inspection efficiency and ensure the safety of aviation operation. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0020] Figure 1 is a schematic diagram of the arrangement of the four-camera array provided by the embodiments of the present application; Figure 2 is a schematic diagram of the scene of image transmission between the four-camera array and the electronic device provided by the embodiments of the present application; Figure 3is a flowchart of an image registration method provided by an embodiment of the present application; Figure 4 is a structural diagram of an image registration device provided by an embodiment of the present application; Figure 5 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the accompanying drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] In order to better understand the embodiments of the present application, the background art will be described in detail as follows: The problem of the traditional image registration method is that speed and accuracy are difficult to achieve. Specifically, although the traditional image registration method (such as a feature matching-based registration method) has certain adaptability to large view angle changes, the calculation load is too high, which leads to low image registration efficiency, and further leads to poor real-time performance of foreign object detection, long single-frame registration time (minute level), and cannot meet the high-frequency detection needs of electronic device (such as airport vehicle detection system) inspection. The mis-matching rate of the optimization algorithm significantly increases under complex road surface environment, especially under large camera view angle difference and rain, fog, and dark light conditions, which leads to insufficient image registration accuracy, and further causes the risk of foreign object missed detection.

[0023] In summary, the traditional image registration method cannot break through the contradiction of "low efficiency - insufficient accuracy", which leads to the inability to efficiently and accurately realize FOD detection, and further seriously restricts the efficiency of airport safety inspection.

[0024] To solve the above technical problems, the embodiment of the present application provides an image registration method, which comprises the following steps: obtaining a target reference image and a first target image to be registered corresponding to a detection area containing foreign objects FOD; constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, N being an integer greater than 1, and performing linear transformation on the first target image to be registered by using the optimal affine transformation model to obtain a second target image to be registered; performing three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image to be registered, and screening extreme points with a principal curvature not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered; using an improved random sample consensus RANSAC algorithm on all first feature points, and screening feature points with a reprojection error and a sampling probability satisfying a preset condition as second feature points; using a singular value decomposition SVD method to solve a homogeneous linear equation set of pixel pairs matched with the second feature points to obtain a transformation homography matrix; and performing registration on the first target image to be registered according to the transformation homography matrix to obtain a target image. The method pre-aligns the image by using the optimal affine transformation model, effectively reduces the computational complexity of the image registration process, reduces the computational load, improves the image registration efficiency, and further shortens the single-frame processing time from minutes to seconds, thereby meeting the real-time requirements of high-frequency detection of electronic equipment. At the same time, the three-dimensional extreme value search of the Gaussian difference pyramid is combined with the principal curvature constraint to effectively filter the edge noise points, and the improved RANSAC algorithm is used to screen the feature points by using the double conditions (reprojection error and sampling probability), thereby effectively reducing the mismatch rate under the conditions of large camera angle difference, rain and fog, and dark light, improving the image registration accuracy, and further reducing the risk of foreign object missed detection. That is, the method can break through the contradiction of "low efficiency and insufficient accuracy", and can efficiently and accurately realize FOD detection in the future, thereby improving the airport safety inspection efficiency and ensuring the safety of aviation operation.

[0025] In order to better understand the embodiments of the present application, the multi-view vision system in which the multi-view camera array involved in the embodiments of the present application is described in detail as follows: I. Hardware architecture of multi-view vision system The hardware architecture of the multi-view vision system mainly comprises an optical sensing unit, an ambient light regulation unit and a trigger control component. The optical sensing unit adopts a multi-view camera array and a supporting mechanical structure to realize vision information acquisition, i.e. image acquisition; the ambient light regulation unit realizes illumination compensation by using an adjustable light source; and the trigger control component is equipped with a programmable timing signal to realize image synchronous acquisition of the multi-view camera.

[0026] Optionally, the core functions of the multi-view vision system described above can include: completing real-time synchronization capture of multiple optical signals based on a hardware synchronization mechanism, optimizing environmental lighting conditions using active light compensation technology to improve image acquisition adaptability, and realizing timing control of multi-device collaborative work through an external trigger interface, etc.

[0027] It should be noted that the optical sensing unit described above generally adopts a global shutter complementary metal-oxide-semiconductor (CMOS) sensor array to construct a multi-view imaging system (i.e., a multi-view camera array). Optionally, the cameras in the multi-view camera array are all high-speed industrial cameras, which can obtain rich information of the detection area through multi-view images for subsequent image registration. In addition, the gimbal support matched in the optical sensing unit supports multi-degree-of-freedom precision adjustment to meet the attitude calibration requirements in complex installation scenarios.

[0028] For example, in the case of a multi-view camera array being a four-view camera array, the arrangement of the four-view camera array is as shown in Figure 1 As can be seen from Figure 1 , the four-view camera array includes four cameras, namely camera C1, camera C2, camera C3, and camera C4. The four cameras are diagonally arranged at a preset angle to capture the detection area, which can maximize the three-dimensional feature expression of the foreign object FOD while ensuring field of view coverage.

[0029] In combination with Figure 1 , as shown in Figure 2 , is a scene schematic diagram of image transmission between the four-view camera array and the electronic device provided by the embodiments of the present application. As can be seen from Figure 2 , after the four-view camera array described above captures images of the detection area, it can transmit all the captured images to the electronic device through a switch using a gigabit network cable connection. The electronic device receives all the images through a switch using a gigabit network cable connection, in preparation for subsequent image registration.

[0030] II. Software architecture of multi-view vision system The software architecture of the multi-view vision system is developed based on Python3.8.19, and the core functions are realized by key libraries. Specifically, OpenCV3.4.10.35 is responsible for bottom-layer image acquisition, processing, and real-time operation; Scikit-image0.21.0 provides high-level image analysis algorithms (such as segmentation, morphological processing, etc.); Scikit-learn1.3.2 supports machine learning tasks; Numpy1.24.4 is used as a core data carrier to realize efficient data exchange between libraries. The multi-view vision system uses Joblib1.3.2 to realize parallel computing acceleration.

[0031] The user interaction layer adopts the PyQt5 5.15.10 framework for software development, and a graphical user interface (GUI) is built by Qt Designer to realize parameter control, data input, and result visualization. The developed software runs under the Windows 10 system, relies on an i7-12700K processor and a 64 GB memory hardware platform, and provides efficient and stable image processing and intelligent analysis capabilities for the half-group micro dual-in-line system of electronic equipment.

[0032] The following describes the execution subject involved in the embodiments of the present application in detail: The execution subject involved in the embodiments of the present application can be an image registration device, or an electronic device. Optionally, the electronic device can include a control terminal, an airport vehicle-mounted detection system, a computer, a mobile terminal, a wearable device, and the like.

[0033] The following describes the image registration method provided by the embodiments of the present application in detail by taking an electronic device as an example: Figure 3 FIG. 1 is a flowchart of the image registration method provided by the embodiments of the present application. As shown in FIG. 1, the method includes the following steps 301-304. Figure 3

[0034] Step 301: Obtain a target reference image and a first target image to be registered corresponding to a to-be-inspected region containing foreign object debris (FOD).

[0035] The target reference image is used as a reference frame for registration, and is used to adjust the spatial alignment of the first target image to be registered.

[0036] The first target image to be registered refers to a to-be-registered image corresponding to the target reference image and subjected to the same preprocessing. The number of the first target image to be registered is at least one.

[0037] It should be noted that the timing sequence in which the electronic device obtains the target reference image and the first target image to be registered is not limited.

[0038] The following describes in detail how the electronic device obtains the target reference image and the first target image to be registered corresponding to the to-be-inspected region containing foreign object debris (FOD): ​In some embodiments, the electronic device obtaining the target reference image and the first target image to be registered corresponding to the area to be inspected containing the foreign object FOD can include: the electronic device obtaining images collected by the multi-view camera array on the area to be inspected; the electronic device taking the image collected by one camera in the multi-view camera array as the first reference image, and taking the images collected by the remaining cameras as the first image to be registered; the electronic device respectively performing image scaling on the first reference image and the first image to be registered through a preset sampling operator to obtain a second reference image and a second image to be registered; and the electronic device respectively performing high-frequency noise elimination on the second reference image and the second image to be registered through a Gaussian filter to obtain the target reference image and the first target image to be registered.

[0039] The preset sampling operator is an image scaling algorithm that generates a thumbnail by reducing the image resolution, which can reduce the calculation amount of subsequent image registration processing while retaining key features.

[0040] The Gaussian filter is a linear smoothing filter, which is a weighted smoothing operation based on a Gaussian kernel. The Gaussian function is used to weight and average the image pixels, thereby attenuating and scaling the high-frequency noise such as jaggies and speckles generated in the process, and improving the robustness of the image registration process.

[0041] In the embodiments of the present application, after the multi-view camera array collects images of the area to be inspected, it can send all the collected images to the electronic device. After receiving all the images, the electronic device can take the image collected by any one camera in the multi-view camera array as the first reference image, and take the images collected by the remaining cameras in the multi-view camera array as the first image to be registered. It can be understood that, in combination with Figure 1 If the electronic device takes the image collected by camera C1 as the first reference image, it will take the images collected by cameras C2, C3 and C4 as the first image to be registered. If the electronic device takes the image collected by camera C2 as the first reference image, it will take the images collected by cameras C1, C3 and C4 as the first image to be registered. Herein, no specific limitation is made.

[0042] Then, the electronic device reduces the sampling of the first reference image and the first image to be registered by combining quadratic sampling with Gaussian filtering, that is, the reduction of the sampling refers to reducing the image size by a fixed ratio, and at the same time using a Gaussian filter to eliminate the speckles generated during the scaling. The specific process is as follows: In the process of reducing the sampling of the first reference image, the electronic device can perform image scaling on the first reference image through a preset sampling operator to obtain a second reference image. The calculation formula of the second reference image is as follows: wherein, represents the second reference image, represents the preset sampling operator, denotes the first reference image; further, the electronic device performs high-frequency noise elimination on the second reference image by a Gaussian filter to obtain a target reference image, and a calculation formula of the target reference image is: wherein, denotes the target reference image, denotes the Gaussian filter.

[0043] In the process of down-sampling the first to-be-registered image, the electronic device can perform image scaling on the first to-be-registered image by the preset sampling operator to obtain a second to-be-registered image, and a calculation formula of the second to-be-registered image is: wherein, denotes the second to-be-registered image, denotes the first to-be-registered image; further, the electronic device performs high-frequency noise elimination on the second to-be-registered image by the Gaussian filter to obtain a target to-be-registered image, and a calculation formula of the target to-be-registered image is: wherein, denotes the target to-be-registered image.

[0044] Exemplarily, it is assumed that the preset sampling operator is a sampling operator with a sampling factor , and the Gaussian filter is a Gaussian filter with a standard deviation of 1.5. Taking a first reference image with a resolution of 800x800 pixels as an example, the resolution of a target reference image obtained after down-sampling is reduced to 267x267 pixels, the data amount of the target reference image is reduced to 1 / 9 of the first reference image, the Gaussian filter adopts a 9x9 kernel function, eliminates high-frequency noise caused by pixel resampling in the process of down-sampling, and retains structural features greater than 10 pixels (corresponding to foreign matters greater than 5 mm in the airport runway) in the target reference image, thereby laying a foundation for a subsequent image registration process.

[0045] It should be noted that, by using the down-sampling-filter joint preprocessing technology, the electronic device can compress the image data amount to 1 / 9 of the original image on the premise of retaining the key features of foreign matters greater than 5 mm, thereby providing engineering feasibility for embedded device deployment.

[0046] Step 302, constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, and performing linear transformation on the first target to-be-registered image by using the optimal affine transformation model to obtain a second target to-be-registered image.

[0047] wherein, N is an integer greater than 1.

[0048] The preset pitch angles are used to describe the lifting or lowering state of the camera lens relative to the horizontal direction, specifically referring to the inclination angle of the camera in the vertical plane (i.e., the front-back direction), usually taking the horizontal plane as the reference (0°), with positive angles upward and negative angles downward. It should be noted that any two of the N preset pitch angles are different.

[0049] The affine transformation model is a linear transformation model for describing the geometric deformation of a two-dimensional plane. It maps images with different viewing angles to a unified geometric space by simulating camera pitch, rotation, and other attitude changes. The affine transformation model includes translation, rotation, scaling, and shearing operations to achieve attitude normalization of multi-view images. The above optimal affine transformation model specifically makes the alignment effect of the target reference image and the first target image to be registered best.

[0050] In step 302, the electronic device adopts an extended SIFT feature matching algorithm, namely the Affine Scale-Invariant Feature Transform (ASIFT) algorithm, to solve the image deformation problem caused by large viewing angle differences. The ASIFT algorithm generates multiple sets of affine transformed image sequences by simulating the imaging process of the camera at different preset pitch angles and optimal azimuth angles, and then extracts feature points with affine invariance. Specifically, the electronic device constructs N affine transformation models according to the target reference image and N preset pitch angles, selects the optimal affine transformation model from the N affine transformation models, and then uses the optimal affine transformation model to perform linear transformation on the first target image to be registered, obtaining the second target image to be registered with the best transformation effect, for subsequent accurate image registration. The entire process uses a cooperative sampling mechanism that proposes a preset pitch angle constraint range and an optimal azimuth angle dynamic interval strategy, significantly reduces redundant calculations through parameter space compression, effectively reduces the computational complexity of the image registration process, and reduces the computational load to improve the image registration efficiency. In turn, the single-frame processing time is shortened from minutes to seconds, meeting the real-time requirements of high-frequency detection of the electronic device.

[0051] The following describes in detail how the electronic device constructs the optimal affine transformation model according to the target reference image and N preset pitch angles: In some embodiments, the electronic device constructs an optimal affine transformation model according to the target reference image and N preset pitch angles, which can include: the electronic device obtains N preset pitch angles from a preset pitch angle range; the electronic device constructs N affine transformation models according to the N preset pitch angles and the respective corresponding optimal azimuth angles; the electronic device respectively adopts the N affine transformation models for feature point detection on the target reference image, to obtain the feature point quantity and the spatial distribution entropy corresponding to each affine transformation model; the electronic device determines n affine transformation models from the N affine transformation models in descending order of the feature point quantity, 0 < n < N; and the affine transformation model with the maximum spatial distribution entropy in the n affine transformation models is taken as the optimal affine transformation model.

[0052] The preset pitch angle range is used to limit the pitch movement range of the multi-camera array in actual operation, to ensure that all possible viewing angles of the to-be-inspected region are covered. Specifically, it refers to the interval of the allowed tilt angle of the multi-camera array in the vertical plane, which is usually based on the horizontal direction, and the positive and negative boundary values are respectively set for the upward (head up) and downward (head down).

[0053] The azimuth angle refers to the rotation angle of the camera in the horizontal plane (0° with the north or reference direction as the reference, increasing clockwise). The above optimal azimuth angle is the best horizontal rotation angle obtained by calibration or optimization for a certain preset pitch angle, so that the matching degree of the camera viewing angle and the target reference image is the highest.

[0054] The above spatial distribution entropy is an index for measuring the uniformity of the distribution of feature points in the image. The greater the spatial distribution entropy, the more uniform the distribution of feature points. In addition, the introduction of the spatial distribution entropy effectively reduces the interference of local noise or abnormal feature points on the selection of the optimal affine transformation model.

[0055] In the embodiments of the present application, in view of the camera view angle difference problem in the multi-camera array, since the preset pitch angle range (such as (45°, 70°)) can cover a large part of the actual shooting angle, the electronic device can limit the N preset pitch angles within the preset pitch angle range for affine transformation simulation. Based on this, the electronic device can first randomly select N preset pitch angles from the preset pitch angle range, and determine the optimal azimuth angle corresponding to each of the N preset pitch angles. In this way, a preset pitch angle and an optimal azimuth angle corresponding thereto can jointly determine a spatial pose (pitch + horizontal rotation) of a camera. The spatial pose is an affine transformation model. In this way, the electronic device can finally determine N affine transformation models. The entire process covers various view angle changes that may exist due to foreign objects by combining the preset pitch angle range and the optimal azimuth angle, thereby avoiding matching failure that may be caused by a single affine transformation model. Then, the electronic device performs feature point detection on the target reference image by using the N affine transformation models respectively, and counts the number of feature points and the spatial distribution entropy corresponding to each affine transformation model. In this way, the electronic device can finally determine N numbers of feature points and N spatial distribution entropies. Next, the electronic device sorts the N numbers of feature points in descending order, and selects the first n numbers of feature points. The entire process preliminarily selects the first n affine transformation models with the largest number of feature points, thereby ensuring sufficient matching point support for transformation calculation and avoiding registration failure caused by too few feature points. Finally, the electronic device determines the maximum spatial distribution entropy from the spatial distribution entropies corresponding to the first n affine transformation models respectively, and then takes the affine transformation model corresponding to the maximum spatial distribution entropy as the optimal affine transformation model. The entire process selects the affine transformation model with the most uniform feature point distribution as the optimal affine transformation model on the basis of meeting the feature point quantity requirement, thereby preventing feature points from being concentrated in a local region (such as a part with rich texture), and improving the global alignment accuracy.

[0056] It should be noted that, in view of the large view angle registration problem of the electronic device, the downsampling constraint parameter compression technology is adopted in the feature point matching link. Specifically, the calculation load is reduced by 3 times image thumbnail preprocessing, the effective range of the preset pitch angle of the camera is locked to 45° to 70°, the traditional full-angle redundant sampling is broken through, the affine transformation simulation times are sharply reduced from 61 times of traditional full-angle sampling to 24 times in combination with the azimuth angle dynamic interval strategy, the single-frame registration time is reduced by 3 times image thumbnail preprocessing, the operation efficiency is greatly improved by 5 times, and the real-time detection demand of the electronic device is completely met.

[0057] In some embodiments, the electronic device constructs N affine transformation models according to the N preset pitch angles and the optimal azimuth angles corresponding thereto, which can include: for each preset pitch angle, the electronic device determines an inclination parameter according to the preset pitch angle, and the calculation formula of the inclination parameter is: , denotes a tilt parameter, denotes a preset pitch angle; the electronic device determines an azimuth sampling interval according to the tilt parameter, and the calculation formula of the azimuth sampling interval is: , denotes an azimuth sampling interval, denotes a constant; the electronic device determines an optimal azimuth according to the azimuth sampling interval; and the electronic device constructs an affine transformation model according to the optimal azimuth and the preset pitch angle.

[0058] The tilt parameter is a scale factor, which is used to quantify the influence of the preset pitch angle on the geometric deformation of the target reference image.

[0059] The azimuth sampling interval is an angle difference between adjacent sampling azimuths in the horizontal plane.

[0060] Since the determination process of each affine transformation model is the same, the electronic device can perform the following operations for each preset pitch angle through a parameterized sampling strategy: since the sampling density in the horizontal direction (azimuth) needs to be adjusted according to the tilt parameter to maintain the uniformity of feature matching, the electronic device can calculate the tilt parameter by bringing the preset pitch angle into the calculation formula of the tilt parameter; and since the larger the tilt parameter (i.e., the larger the preset pitch angle), the smaller the azimuth sampling interval , i.e., the more dense the sampling in the horizontal direction, to compensate for the feature sparsification caused by the pitch deformation, a constant = 72° can be introduced to balance the sampling density and the calculation efficiency, based on which the electronic device calculates the azimuth sampling interval by bringing the tilt parameter into the calculation formula of the azimuth sampling interval; then, the electronic device samples the range of (0°, 360°) according to the azimuth sampling interval to obtain a plurality of azimuths, and determines an optimal azimuth from the plurality of azimuths, and further constructs an affine transformation model in combination with the preset pitch angle. In this way, the electronic device can finally determine N affine transformation models.

[0061] Exemplarily, assuming that N is 2, the two affine transformation models are affine transformation model 1 and affine transformation model 2, wherein the affine transformation model 1 is ( = 45°, = 45°), and the affine transformation model 2 is ( = 60°, = 36°).

[0062] In step 303, the electronic device performs a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, filters extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered, and uses an improved random sample consensus (RANSAC) algorithm to filter feature points with a reprojection error and a sampling probability satisfying a preset condition as second feature points.

[0063] The Gaussian difference (DoG) pyramid is a multi-scale image representation constructed by differencing Gaussian blurred versions of the same image (i.e., the second target image to be registered) at different scales.

[0064] The principal curvature is an index for describing the maximum bending degree and the minimum bending degree of a curved surface at an extreme value point, and is used to describe the geometric shape characteristics of a local curved surface (or an image intensity surface) where a key point (extreme value point) is located. The main purpose is to filter out points with strong edge responses (such as edge points) and retain stable corner or spot features.

[0065] The reprojection error is a key index for evaluating the matching quality of feature points and the accuracy of a geometric model.

[0066] The sampling probability is the probability weight of each feature point being selected to participate in model estimation in the improved random sample consensus (RANSAC) algorithm. The RANSAC algorithm is a robust model fitting method that filters the optimal model from noisy data through iterative sampling.

[0067] In step 303, after determining the Gaussian difference pyramid corresponding to the second target image to be registered, the electronic device performs a three-dimensional extreme value search on the Gaussian difference pyramid, performs a first screening of extreme value points based on principal curvatures to eliminate edge-sensitive points and retain robust features, obtains first feature points with better quality, and the number of first feature points is multiple. Then, the electronic device uses an improved RANSAC algorithm to perform a second screening of all first feature points based on a reprojection error and a sampling probability, avoids the blind randomness of a traditional RANSAC, accelerates convergence, obtains second feature points with better accuracy, and is ready for subsequent accurate registration of images. The number of second feature points is also multiple.

[0068] Optionally, before step 303, the method may further include: the electronic device using a Gaussian kernel with M+1 standard deviations to blur the second target image to be registered, respectively, to obtain M+1 Gaussian images, where M is an integer greater than or equal to 3; the electronic device subtracting two adjacent Gaussian images in the M+1 Gaussian images to construct a Gaussian difference pyramid, which includes M Gaussian difference images.

[0069] For example, if M=4, then the standard deviation The quantity is 5, respectively =1.0、 =1.4、 =2.0、 =2.4 and =4.8. Based on this, the electronic device can use these 5 standard deviations of Gaussian kernels to blur the second target image to be registered, generating 5 Gaussian images with different degrees of blurring. This can highlight those features that are obvious under different scaling degrees. Then, the electronic device subtracts the two adjacent Gaussian images in the above 5 Gaussian images to obtain 4 Gaussian difference images to construct a Gaussian difference pyramid.

[0070] With the first one standard deviation and the +1 standard deviation For example, > , ={1,…,M}, electronic devices use standard deviation The Gaussian kernel is used to blur the second target image to be registered, generating a Gaussian image α, and the standard deviation is used. The Gaussian kernel blurs the second target image to be registered, generating a Gaussian image β, where Gaussian image β and the aforementioned Gaussian image α are two adjacent Gaussian images. Then, the electronic device subtracts Gaussian image β from Gaussian image α to obtain a difference-of-Gaussian image. The calculation formula for the difference-of-Gaussian image is as follows: .

[0071] in, Represents the difference-of-Gaussian image, ( () represents the pixel coordinates at the corresponding position; β represents the Gaussian image. = , Standard deviation Gaussian kernel, This represents the image to be registered for the second target. Indicates the convolution operation; Let α represent the Gaussian image. = 2 , denotes a standard deviation of a Gaussian kernel.

[0072] The electronic device performs a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and details are as follows: the electronic device filters extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered: In some embodiments, the Gaussian difference pyramid includes M Gaussian difference images, M is an integer greater than 2, and the M Gaussian difference images include a target Gaussian difference image, the target Gaussian difference image being a Gaussian difference image adjacent to two Gaussian difference images.

[0073] Based on this, the electronic device performs a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and filters extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered, which can include: traversing each candidate pixel point in the target Gaussian difference image, the electronic device obtains a first pixel point sharing a corner point with the candidate pixel point; from the two Gaussian difference images adjacent to the target Gaussian difference image, the electronic device determines a second pixel point corresponding to the positions of the candidate pixel point and the first pixel point; the pixel point with the maximum or minimum brightness value among the candidate pixel point, the first pixel point and the second pixel point is taken as an extreme value point; the principal curvature is determined according to the Hessian matrix of the extreme value point; the electronic device filters extreme value points with principal curvatures not exceeding a preset principal curvature threshold from all extreme value points as first feature points of the second target image to be registered.

[0074] The Hessian matrix can also be called Hesse matrix or Haise matrix, which is a square matrix composed of second-order partial derivatives of a multivariate function, and is used to describe the curvature characteristics of local structure in image processing.

[0075] In the process of performing the first screening, a classical feature extraction algorithm (such as a Scale-Invariant Feature Transform (SIFT) algorithm) can be used, and the core idea is to use the extreme points in the scale space detected by a Gaussian difference pyramid as candidate pixel points, which can also be referred to as candidate feature points, so that the finally determined first feature points have scale, rotation and illumination invariance. Specifically, the electronic device first determines a Gaussian difference pyramid corresponding to the second target image to be registered, and determines a target Gaussian difference image from M Gaussian difference images included in the Gaussian difference pyramid; then, the electronic device traverses each candidate pixel point in the target Gaussian difference image, and obtains first pixel points that share a corner point with the candidate pixel point. It can be understood that if the candidate pixel point is a corner point, the number of first pixel points is 3, if the candidate pixel point is an edge point, the number of first pixel points is 5, and if the candidate pixel point is neither a corner point nor an edge point, the number of first pixel points is 8; next, the electronic device determines two Gaussian difference images adjacent to the target Gaussian difference image from the M Gaussian difference images included in the Gaussian difference pyramid, and determines second pixel points corresponding to the candidate pixel point and the first pixel points from the two Gaussian difference images; subsequently, the electronic device takes the pixel point with the maximum (i.e., the brightest) or minimum (i.e., the darkest) brightness value among the candidate pixel point, the first pixel point and the second pixel point as an extreme point, and calculates the Hessian matrix of the extreme point to determine the principal curvature of the extreme point; finally, the electronic device further screens the extreme points that are not very sensitive to changes in viewing angle from all the extreme points, i.e., screens the extreme points with a principal curvature not exceeding a preset principal curvature threshold as the first feature points of the second target image to be registered.

[0076] For example, assuming the candidate pixel is a corner point, the surrounding pixels of the corner point are 2×2 pixels. These 2×2 pixels include the corner point and the first pixel sharing the corner point. In this case, the electronic device determines the second pixel corresponding to the position of the 2×2 pixels from two adjacent Gaussian difference images, and then takes the brightest or darkest pixel in the 2×2×3 range as the extreme point. Alternatively, assuming the candidate pixel is an edge point, the surrounding pixels of the edge point are 2×3 pixels. These 2×3 pixels include the edge point and the first pixel sharing the corner point. In this case, the electronic device determines the second pixel corresponding to the position of the 2×2 pixels from two adjacent Gaussian difference images, and then takes the brightest or darkest pixel in the 2×2×3 range as the extreme point. The second pixel corresponding to the position of 2×3 pixels is determined from two adjacent Gaussian difference images. Then, the brightest or darkest pixel in the 2×3×3 range is taken as the extreme point. Assuming that the above candidate pixel is neither a corner point nor an edge point, the surrounding pixels of the candidate pixel are 3×3 pixels. These 3×3 pixels include the candidate pixel and the first pixel that shares a corner point with the candidate pixel. At this time, the electronic device determines the second pixel corresponding to the position of 3×3 pixels from two adjacent Gaussian difference images. Then, the brightest or darkest pixel in the 3×3×3 range is taken as the extreme point.

[0077] Optionally, the above Hessian matrix is .

[0078] in, Indicates the extreme point at The first and second partial derivatives in the direction; Indicates the extreme point at The second partial derivative in the direction, This represents the cross-partial derivative of the first and second order partial derivatives and the second and second order partial derivatives.

[0079] In some embodiments, the electronic device determines the principal curvature based on the Hessian matrix of the extreme points, which may include: the electronic device determining the principal curvature based on the extreme points at... The first and second partial derivatives in the direction, and the extreme points are at The Hessian matrix is ​​constructed using the second and second-order partial derivatives of the directions, and the cross-partial derivatives of the first and second-order partial derivatives. Based on the Hessian matrix, the electronic device determines the principal curvature, and the formula for calculating the principal curvature is: .

[0080] in, Indicates the principal curvature; Represents the Hessian matrix; This represents the summation of the first and second-order partial derivatives, i.e. ; This represents the difference between the product of the first and second-order partial derivatives and the square of the cross partial derivatives, i.e. .

[0081] For example, assuming that the preset principal curvature threshold is set to 100, if the principal curvature is greater than 100, it indicates that the extreme point has a great difference in principal curvature in two directions, the extreme point is sensitive to the change of the viewing angle, and is not an edge point of the foreign object, and thus the extreme point needs to be removed; if the principal curvature is less than or equal to 100, it indicates that the extreme point has a small difference in principal curvature in two directions, the extreme point is not very sensitive to the change of the viewing angle, and is an edge point of the foreign object, and thus the extreme point can be retained.

[0082] The electronic device uses the improved RANSAC algorithm on all the first feature points, and the details of screening the feature points with the re-projection error and the sampling probability satisfying the preset condition as the second feature points are as follows: In some embodiments, the electronic device uses the improved RANSAC algorithm on all the first feature points, and screens the feature points with the re-projection error and the sampling probability satisfying the preset condition as the second feature points, which can include: the electronic device randomly selects P1 first sub-feature points and P2 second sub-feature points from all the first feature points, P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of the first feature points; the electronic device calculates the homography matrix according to the P1 first sub-feature points; for each second sub-feature point, the electronic device determines the re-projection error according to the homography matrix, the pixel coordinates of the second sub-feature point, and the pixel coordinates of the pixel point corresponding to the second sub-feature point in the second target image to be registered; and determines the sampling probability according to the re-projection error; and the electronic device screens the second sub-feature points with the re-projection error less than the preset error threshold and the sampling probability greater than the preset probability threshold from the P2 second sub-feature points as the second feature points.

[0083] wherein the above homography matrix is a 3x3 homogeneous matrix, which is a mapping matrix used to describe the perspective transformation relationship between two plane images, and can be represented as .

[0084] and represent the scaling factor of the camera; and represent the shearing transformation affecting the image; and represent the translation component; and represent the perspective distortion correction; represent the homogeneous coordinate normalization factor, which is usually set to 1.

[0085] In the process of secondary screening, the improved RANSAC algorithm can be used to screen all the first feature points, specifically, the electronic device screens the first feature points in the first image and the second feature points in the second image in the first screening process, and then screens the first feature points in the first image and the second feature points in the second image in the second screening process. ​​P1 first sub-feature points and P2 second sub-feature points are randomly selected in the next iteration, and a homography matrix corresponding to the P1 first sub-feature points is determined, and the value of P1 is usually 4. Then, the geometric consistency is verified for the first second sub-feature point in the P2 second sub-feature points. Specifically, the electronic device determines the re-projection error of the first second sub-feature point according to the homography matrix, the pixel coordinates of the first second sub-feature point, and the pixel coordinates of the pixel point corresponding to the first second sub-feature point in the second target image to be registered. Then, the electronic device determines the initial sampling probability of the first second sub-feature point according to the re-projection error, and further calculates the sampling probability. Based on this, the electronic device can finally determine the re-projection error and the sampling probability of each of the P2 second sub-feature points. Finally, the electronic device screens the second sub-feature points with a re-projection error less than a preset error threshold (such as 1.5) and a sampling probability greater than a preset probability threshold (such as 0.9) from all first feature points as second feature points. The calculation formula of the re-projection error is: ,

[0086] wherein L2 norm, i.e., Euclidean norm. The calculation formula of the initial sampling probability is: .

[0087] wherein,

[0088] indicates the weight of the first second sub-feature point in the i-th iteration; indicates the weight of the first second sub-feature point in the i-th iteration; indicates the weight decay factor based on the Gaussian kernel, and exemplarily, the weight is iterated 10 times. The calculation formula of the sampling probability is: .

[0089] indicates the sampling probability of the first second sub-feature point;

[0090] indicates the sum of the sampling probabilities of all second sub-feature points.

[0091] ​​​​It should be noted that, for the large angle registration problem of the electronic device, a double verification mechanism is developed in the false matching suppression link, specifically a hypothesis filtering process of 4-point modeling + 1-point pre-check in 5-point sample set is adopted, the geometric consistency is verified by the 5th sample point in each RANSAC iteration to quickly filter out the wrong model, and the dynamic weight adjustment based on the residual is matched, which significantly reduces the false matching interference in complex environment, finally realizes high-precision registration, meets the real-time detection demand of the airport vehicle-mounted system, and significantly suppresses the complex runway texture interference, even in the condition of large angle difference (> 60°) and rain, fog and dark light, the false matching rate is still stable, and the foreign matter missed detection risk is reduced compared with the traditional image registration method.

[0092] Optionally, in the process of screening all the first feature points by using the improved RANSAC algorithm, the electronic device can calculate the spatial dispersion of the sample point set (i.e. all the first feature points); if the spatial dispersion is less than a preset dispersion threshold (such as 0.3), resampling is triggered to obtain new second feature points, ensuring uniform distribution of the point set.

[0093] Among them, the spatial dispersion is used to measure the uniformity of the distribution of the sample point set, the greater the spatial dispersion, the more uneven the distribution of the sample point set (may be concentrated in a local or have outliers), and the closer the spatial dispersion is to 0, the more uniform the distribution of the sample point set (i.e. the maximum value and the minimum value are close to the average value).

[0094] Optionally, the calculation formula of the spatial dispersion is: .

[0095] X represents the horizontal coordinate component of all points in the sample point set; Y represents the vertical coordinate component of all points in the sample point set; Max represents the maximum value of all vertical coordinate components and all horizontal coordinate components; Min represents the minimum value of all vertical coordinate components and all horizontal coordinate components; Mean represents the arithmetic mean of all vertical coordinate components and all horizontal coordinate components.

[0096] It should be noted that the dynamic probability updating method based on spatial dispersion and residual weighting can effectively ensure the uniformity of the distribution of the feature points.

[0097] Optionally, after step 303, the method may further include: performing the following operations for each second feature point: the electronic device calculates a gradient direction histogram within a preset pixel neighborhood (e.g., a 16×16 pixel neighborhood) centered on the second feature point, and determines the main direction, which is the mode of gradient directions within the preset pixel neighborhood (i.e., the direction with the highest frequency of occurrence). In addition, when the secondary peak exceeds 80% of the main peak, an auxiliary direction is generated, i.e., an additional descriptor is generated to enhance rotation robustness; the electronic device divides the preset pixel neighborhood into neighborhood sub-blocks (e.g., 4×4 neighborhood sub-blocks), and calculates 8 directional gradient histograms for each neighborhood sub-block, concatenating them into a 128-dimensional feature vector. After L2 normalization, a descriptor robust to rotation, scaling, and illumination is obtained for the second feature point. The goal of normalization is to eliminate the effects of illumination changes (e.g., linear changes in brightness do not alter the descriptor) and improve robustness to local contrast. Based on this, the electronic device can ultimately determine the descriptor for each second feature point. In this way, even with changes in image brightness, slight rotation, or scale, the descriptors for the same feature point will be highly similar, providing a precise basis for subsequent image registration.

[0098] The 128-dimensional dimension is calculated from 16 neighboring sub-blocks × 8 directions of each neighboring sub-block.

[0099] Optionally, the formula for calculating the above descriptor is: .

[0100] Descriptor representing the second feature point (normalized to unit length); Representing the eigenvector The Gradient magnitude in each dimension ( (∈[1,128]), calculated from the horizontal and vertical gradients of the image; Representing the eigenvector The L2 norm.

[0101] Step 304: Solve the homogeneous linear equations of the pixel pairs that match the second feature point using the Singular Value Decomposition (SVD) method to obtain the transformation homography matrix; and register the first target image to be registered according to the transformation homography matrix to obtain the target image.

[0102] Among them, the Singular Value Decomposition (SVD) method is a mathematical method that decomposes any real or complex matrix into the product of three specific matrices.

[0103] In step 304, the electronic device solves the homographic matrix by SVD method, effectively eliminates the noise and outliers in feature point matching, improves the geometric accuracy of image registration, and then realizes efficient alignment of the first target image to be registered and the target reference image based on the global projection transformation of the homographic matrix, ensures stable registration results under the difference of light and angle, and obtains target images with better quality.

[0104] The electronic device uses the singular value decomposition (SVD) method to solve the homogeneous linear equation set of the pixel point pair matched with the second feature point, and the detailed description of obtaining the transformation homographic matrix is as follows: In some embodiments, the electronic device uses the singular value decomposition (SVD) method to solve the homogeneous linear equation set of the pixel point pair matched with the second feature point, and the detailed description of obtaining the transformation homographic matrix is as follows:

[0105] In the process of determining the transformation homographic matrix, for each second feature point in the S second feature points, S is an integer greater than 1, the electronic device determines the feature point closest to the descriptor of the second feature point from the target reference image as the pixel point matched with the second feature point, and constructs the second feature point and the pixel point as a pixel point pair, taking the pixel coordinates of the i-th second feature point and the matched pixel point as an example. The equation is expanded as follows: and the equation is rewritten as a matrix equation: wherein, , represents a flattened vector constructed by all elements in the transformation homographic matrix.

[0106] For S second feature points, the electronic device can construct S pixel point pairs, and then construct S homogeneous equations to obtain a homogeneous linear equation set; finally, the electronic device uses the SVD method to solve the homogeneous linear equation set to obtain the transformation homographic matrix, which is ready for subsequent accurate registration of the first target image to be registered.

[0107] The electronic device uses the singular value decomposition (SVD) method to solve the homogeneous linear equation set of the pixel point pair matched with the second feature point, and the detailed description of obtaining the transformation homographic matrix is as follows: In some embodiments, the above homogeneous linear equation set is , , represents the coefficient matrix.​​

[0108] Based on this, the electronic device solves the homogeneous linear equation set constructed by all homogeneous equations by using the SVD method to obtain the transformation homography matrix, which can include: the electronic device decomposes the coefficient matrix by using the SVD method to obtain the right singular vector matrix ; the electronic device takes the last column element in the right singular vector matrix as a flattened vector ; and the electronic device constructs the transformation homography matrix according to all elements in the flattened vector .

[0109] In the process of constructing the transformation homography matrix, since the electronic device has constructed S matrix forms, the electronic device constructs a 2Sx9 coefficient matrix for the S pixel pairs, and solves the above homogeneous linear equation set by using the SVD method. Specifically, the electronic device decomposes the coefficient matrix to obtain , so as to obtain the left singular vector matrix , the diagonal matrix , and the right singular vector matrix ; then the electronic device takes the last column element in the right singular vector matrix as a flattened vector , and further constructs the transformation homography matrix according to all elements in the flattened vector .

[0110] It should be noted that the above coefficient matrix is an m*n matrix; the above left singular vector matrix is an m*m orthogonal matrix; the above diagonal matrix is an m*n diagonal matrix; and the above right singular vector matrix is an n*n orthogonal matrix.

[0111] Optionally, after step 304, the method can further include: the electronic device iterates again by using the improved RANSAC algorithm until the minimized re-projection error is less than a preset re-projection error threshold (such as pixels), at which time it can be determined that the optimal transformation homography matrix is calculated by the optimal single mapping algorithm to achieve accurate registration of the first target image to be registered.

[0112] In the embodiments of the present application, the technical solutions of steps 301-304 are used to pre-align the image through the optimal affine transformation model, effectively reduce the calculation complexity and the calculation load of the image registration process, improve the image registration efficiency, and further shorten the single-frame processing time from minutes to seconds, so as to meet the real-time requirements of high-frequency detection of electronic equipment. At the same time, the three-dimensional extreme value search of the Gaussian difference pyramid combined with the principal curvature constraint is used to effectively filter the edge noise points, and the improved RANSAC algorithm is used to screen the feature points through the double conditions (reprojection error and sampling probability), effectively reduce the mismatching rate under the conditions of large camera view angle difference, rain and fog, and dark light, and improve the image registration accuracy, and further reduce the risk of foreign object detection. That is, this method can break through the contradiction of "low efficiency and insufficient accuracy", and can efficiently and accurately realize FOD detection, and further improve the safety inspection efficiency of the airport, and ensure the safety of the aviation operation.

[0113] The image registration device provided by the embodiments of the present application is described below. The image registration device described below can be correspondingly referred to the image registration method described above.

[0114] Figure 4 is a structural schematic diagram of the image registration device provided by the embodiments of the present application. As shown in Figure 4 , the device comprises an acquisition module 401, a transformation module 402, a screening module 403 and a registration module 404.

[0115] The acquisition module 401 is configured to acquire a target reference image and a first target image to be registered corresponding to a detection area containing foreign objects FOD. The transformation module 402 is configured to construct an optimal affine transformation model according to the target reference image and N preset pitch angles, N is an integer greater than 1, and perform linear transformation on the first target image to be registered by using the optimal affine transformation model to obtain a second target image to be registered. The screening module 403 is configured to perform three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and screen the extreme value points with the principal curvature not exceeding a preset principal curvature threshold as the first feature points of the second target image to be registered; and use an improved random sample consensus RANSAC algorithm on all first feature points to screen the feature points with the reprojection error and the sampling probability satisfying the preset condition as the second feature points. The registration module 404 is configured to solve the homogeneous linear equation set of the pixel point pair matched with the second feature point by using a singular value decomposition SVD method to obtain a transformation homography matrix; and perform registration on the first target image to be registered according to the transformation homography matrix to obtain a target image.

[0116] Optionally, the transformation module 402 is specifically configured to obtain the N preset pitch angles from a preset pitch angle range; construct N affine transformation models according to the N preset pitch angles and the respective optimal azimuth angles; perform feature point detection on the target reference image by using the N affine transformation models respectively to obtain the number of feature points and the spatial distribution entropy corresponding to each affine transformation model; determine n affine transformation models from the N affine transformation models in descending order of the number of feature points, 0 < n < N; and determine the affine transformation model with the maximum spatial distribution entropy in the n affine transformation models as the optimal affine transformation model.

[0117] Optionally, the Gaussian difference pyramid includes M Gaussian difference images, M is an integer greater than 2, and the M Gaussian difference images include a target Gaussian difference image, the target Gaussian difference image being a Gaussian difference image adjacent to two Gaussian difference images; the screening module 403 is specifically configured to traverse each candidate pixel point in the target Gaussian difference image to obtain a first pixel point sharing a corner point with the candidate pixel point; determine a second pixel point corresponding to the positions of the candidate pixel point and the first pixel point from the two Gaussian difference images adjacent to the target Gaussian difference image; determine a pixel point with the maximum or minimum luminance value among the candidate pixel point, the first pixel point, and the second pixel point as an extreme point; determine the principal curvature according to the Hessian matrix of the extreme point; and screen the extreme points with the principal curvature not exceeding a preset principal curvature threshold value from all the extreme points as the first feature points of the second target image to be registered.

[0118] Optionally, the screening module 403 is specifically configured to randomly select P1 first sub-feature points and P2 second sub-feature points from the all first feature points, P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of the first feature points; calculate a homography matrix according to the P1 first sub-feature points; for each second sub-feature point, determine a re-projection error according to the homography matrix, the pixel coordinates of the second sub-feature point, and the pixel coordinates of the pixel point corresponding to the second sub-feature point in the second target image to be registered; determine a sampling probability according to the re-projection error; and screen the second sub-feature points with the re-projection error less than a preset error threshold value and the sampling probability greater than a preset probability threshold value from the P2 second sub-feature points as the second feature points.

[0119] Optionally, the registration module 404 is specifically configured to determine the pixel points matched with each second feature point from the target reference image and construct pixel point pairs according to the descriptors of all the second feature points; construct homogeneous equations according to the pixel coordinates of each pixel point pair; and solve the homogeneous linear equations constructed by all the homogeneous equations by using the SVD method to obtain the transformation homography matrix.

[0120] Optionally, the transformation module 402 is specifically configured to perform the following operations for each preset pitch angle: determining a tilt parameter according to the preset pitch angle, the calculation formula of the tilt parameter being: , denotes the tilt parameter, denotes the preset pitch angle; determining an azimuth angle sampling interval according to the tilt parameter, the calculation formula of the azimuth angle sampling interval being: , denotes the azimuth angle sampling interval, denotes a constant; determining an optimal azimuth angle according to the azimuth angle sampling interval; and constructing an affine transformation model according to the optimal azimuth angle and the preset pitch angle.

[0121] Optionally, the screening module 403 is specifically configured to construct the Hessian matrix according to a first second-order partial derivative of the extreme point in a direction, a second second-order partial derivative of the extreme point in a direction, and a cross partial derivative of the first second-order partial derivative and the second second-order partial derivative; and determine the principal curvature according to the Hessian matrix, the calculation formula of the principal curvature being: ; wherein, denotes the principal curvature; denotes the Hessian matrix; denotes a summation result of the first second-order partial derivative and the second second-order partial derivative; denotes a difference result between a product result of the first second-order partial derivative and the second second-order partial derivative and a square result of the cross partial derivative.

[0122] Optionally, the homogeneous linear equation set is , denotes a coefficient matrix, denotes a flattened vector constructed by all elements in the transformation homography matrix, and the registration module 404 is specifically configured to decompose the coefficient matrix by using the SVD method to obtain a right singular vector matrix ; take the last column element in the right singular vector matrix as the flattened vector ; and construct the transformation homography matrix according to all elements in the flattened vector .

[0123] Optionally, the acquisition module 401 is specifically configured to acquire images collected by a multi-view camera array on the to-be-inspected region; take an image collected by one camera in the multi-view camera array as a first reference image, and take images collected by the remaining cameras as first to-be-registered images; perform image scaling on the first reference image and the first to-be-registered images respectively through a preset sampling operator to obtain a second reference image and a second to-be-registered image; and perform high-frequency noise elimination on the second reference image and the second to-be-registered image respectively through a Gaussian filter to obtain the target reference image and the first target to-be-registered image.

[0124] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 5 the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute an image registration method, which includes: acquiring a target reference image and a first target to-be-registered image corresponding to a to-be-inspected region containing foreign objects FOD; constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, N being an integer greater than 1, and performing linear transformation on the first target to-be-registered image using the optimal affine transformation model to obtain a second target to-be-registered image; performing three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target to-be-registered image, and screening extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target to-be-registered image; and using an improved random sample consensus RANSAC algorithm on all first feature points to screen feature points with a reprojection error and a sampling probability satisfying a preset condition as second feature points; using a singular value decomposition SVD method to solve a homogeneous linear equation set of pixel point pairs matched with the second feature points to obtain a transformation homography matrix; and performing registration on the first target to-be-registered image according to the transformation homography matrix to obtain a target image.

[0125] Further, the logic instructions in the memory 530 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] In another aspect, the embodiments of the present application also provide a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the image registration method provided by the above-mentioned methods. The method comprises the following steps: acquiring a target reference image and a first target to-be-registered image corresponding to a to-be-inspected area containing foreign objects FOD; constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, N is an integer greater than 1, and performing linear transformation on the first target to-be-registered image by using the optimal affine transformation model to obtain a second target to-be-registered image; performing three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target to-be-registered image, and screening extreme value points with main curvatures not exceeding a preset main curvature threshold as first feature points of the second target to-be-registered image; and using an improved random sample consensus (RANSAC) algorithm on all first feature points, and screening feature points with a reprojection error and a sampling probability satisfying a preset condition as second feature points; using a singular value decomposition (SVD) method to solve a homogeneous linear equation set of pixel point pairs matched with the second feature points to obtain a transformation homography matrix; and performing registration on the first target to-be-registered image according to the transformation homography matrix to obtain a target image.

[0127] In yet another aspect, the embodiments of the present application also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image registration method provided by each of the above methods, and the method comprises: acquiring a target reference image and a first target image to be registered corresponding to a region to be inspected containing foreign objects FOD; constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, N being an integer greater than 1, and performing linear transformation on the first target image to be registered by using the optimal affine transformation model to obtain a second target image to be registered; performing three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image to be registered, and screening extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered; and screening feature points with a reprojection error and a sampling probability satisfying a preset condition as second feature points by using an improved random sample consensus RANSAC algorithm on all first feature points; solving a homogeneous linear equation set of pixel point pairs matched with the second feature points by using a singular value decomposition SVD method to obtain a transformation homography matrix; and performing registration on the first target image to be registered according to the transformation homography matrix to obtain a target image.

[0128] The device embodiments described above are only schematic, wherein the units illustrated as separate components may or may not be physically separate, and the components illustrated as units may or may not be physical units, i.e., may be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0129] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image registration method characterized by, The method comprises the steps of: acquiring a target reference image and a first target image to be registered corresponding to a region to be detected containing foreign object debris (FOD); constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, N being an integer greater than 1, and performing linear transformation on the first target image to be registered by using the optimal affine transformation model to obtain a second target image to be registered; performing three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image to be registered, and screening extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered; and using an improved random sample consensus (RANSAC) algorithm on all first feature points to screen feature points with re-projection errors and sampling probabilities satisfying preset conditions as second feature points; solving a homogeneous linear equation group of pixel pairs matched with the second feature points by using a singular value decomposition (SVD) method to obtain a transformation homography matrix, and performing registration on the first target image to be registered according to the transformation homography matrix to obtain a target image.

2. The image registration method of claim 1, wherein, The method comprises the steps of: acquiring the N preset pitch angles from a preset pitch angle range; constructing N affine transformation models according to the N preset pitch angles and respective optimal azimuth angles; performing feature point detection on the target reference image by using the N affine transformation models respectively to obtain the number and spatial distribution entropy of feature points corresponding to each affine transformation model; determining n affine transformation models from the N affine transformation models in descending order of the number of feature points, 0 < n < N; and taking an affine transformation model with the maximum spatial distribution entropy in the n affine transformation models as the optimal affine transformation model.

3. The image registration method of claim 1, wherein, The Gaussian difference pyramid comprises M Gaussian difference images, M being an integer greater than 2, and the M Gaussian difference images comprise a target Gaussian difference image, the target Gaussian difference image being a Gaussian difference image adjacent to two Gaussian difference images. The method comprises the steps of: traversing each candidate pixel point in the target Gaussian difference image to acquire a first pixel point sharing a corner point with the candidate pixel point; determining a second pixel point corresponding to the positions of the candidate pixel point and the first pixel point from two Gaussian difference images adjacent to the target Gaussian difference image; taking a pixel point with the maximum or minimum luminance value among the candidate pixel point, the first pixel point and the second pixel point as an extreme value point; and determining the principal curvature according to the Hessian matrix of the extreme value point; screening extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered from all extreme value points.

4. The image registration method of claim 1, wherein, The method comprises the steps of: using an improved random sample consensus (RANSAC) algorithm on all first feature points to screen feature points with re-projection errors and sampling probabilities satisfying preset conditions as second feature points. Randomly select P1 first sub-feature points and P2 second sub-feature points from the all first feature points, P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of the first feature points; According to the P1 first sub-feature points, a homography matrix is calculated; For each second sub-feature point, a re-projection error is determined according to the homography matrix, the pixel coordinates of the second sub-feature point, and the pixel coordinates of the pixel point corresponding to the second sub-feature point in the second target image to be registered; and a sampling probability is determined according to the re-projection error; From the P2 second sub-feature points, second sub-feature points with a re-projection error less than a preset error threshold and a sampling probability greater than a preset probability threshold are selected as the second feature points.

5. The image registration method of claim 1, wherein, The singular value decomposition (SVD) method is used to solve the homogeneous linear equations of the pixel point pairs matched with the second feature points to obtain a transformation homography matrix, including: According to the descriptors of all second feature points, pixel points matched with each second feature point are determined from the target reference image, and pixel point pairs are constructed; According to the pixel coordinates of each pixel point pair, a homogeneous equation is constructed; The SVD method is used to solve the homogeneous linear equations constructed by all homogeneous equations to obtain the transformation homography matrix.

6. The image registration method of claim 2, wherein, The N affine transformation models are constructed according to the N preset pitch angles and the optimal azimuth angles corresponding to each preset pitch angle, including: For each preset pitch angle, the following operations are performed: According to the preset pitch angle, a tilt parameter is determined, and a calculation formula of the tilt parameter is: , represents the tilt parameter, represents the preset pitch angle; According to the tilt parameter, a azimuth angle sampling interval is determined, and a calculation formula of the azimuth angle sampling interval is: , denotes the azimuth angle sampling interval, denotes a constant; An optimal azimuth angle is determined according to the azimuth angle sampling interval; An affine transformation model is constructed according to the optimal azimuth angle and the preset pitch angle.

7. The image registration method of claim 3, wherein, The principal curvatures are determined according to the Hessian matrix of the extreme point, including: According to the first second-order partial derivative of the extreme point in the first direction, the second second-order partial derivative of the extreme point in the second direction, and the cross partial derivative of the first second-order partial derivative and the second second-order partial derivative, the Hessian matrix is constructed; According to the Hessian matrix, the principal curvatures are determined, and a calculation formula of the principal curvatures is: ; wherein, denotes the principal curvatures; denotes the Hessian matrix; denotes the sum of the first and second second-order partial derivatives; denotes the difference between the product of the first and second second-order partial derivatives and the square of the cross partial derivative.

8. The image registration method of claim 5, wherein, The homogeneous linear equation system is , denotes the coefficient matrix, denotes a flattened vector built from all elements of the variable transform homography matrix, The SVD method is used to solve the homogeneous linear equations constructed by all homogeneous equations to obtain the transformation homography matrix, including: The SVD method is used to decompose the coefficient matrix to obtain a right singular vector matrix ; the last column element in the right singular vector matrix is taken as the flattened vector ; According to all elements in the flattening vector the warping homography is constructed.

9. The image registration method of any of claims 1-8, wherein, The target reference image and the first target image to be registered corresponding to the foreign object debris (FOD) containing area are obtained, including: Images collected by a multi-view camera array on the area to be inspected are obtained; An image collected by one camera in the multi-view camera array is taken as a first reference image, and images collected by the remaining cameras are taken as first images to be registered; The first reference image and the first image to be registered are respectively scaled by a preset sampling operator to obtain a second reference image and a second image to be registered; The second reference image and the second image to be registered are respectively subjected to high-frequency noise elimination by a Gaussian filter to obtain the target reference image and the first target image to be registered.

10. An image registration apparatus characterized by comprising: It includes: An acquisition module is configured to acquire a target reference image and a first target image to be registered corresponding to a foreign object debris (FOD) containing area to be inspected; A transformation module is configured to construct an optimal affine transformation model according to the target reference image and N preset pitch angles, N is an integer greater than 1, and the optimal affine transformation model is used to perform linear transformation on the first target image to be registered to obtain a second target image to be registered. The screening module is configured to perform a three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image pair to be registered, and screen extreme value points with a principal curvature not exceeding a preset principal curvature threshold as first feature points of the second target image pair to be registered; and adopt an improved random sample consensus (RANSAC) algorithm to screen feature points with a reprojection error and a sampling probability satisfying a preset condition as second feature points. The registration module is configured to adopt a singular value decomposition (SVD) method to solve a homogeneous linear equation set of pixel point pairs matched with the second feature points, and obtain a variable single homographic matrix; and perform registration on the first target image pair to be registered according to the variable single homographic matrix to obtain a target image.

Images