Image registration method and image registration apparatus

Abstract

The application provides an image registration method and an image registration device, and applies to the technical field of image processing. The method comprises the following steps: obtaining a target reference image and a first target image to be registered of a detection area containing FOD; constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, and then performing linear transformation on the first target image to be registered to obtain a second target image to be registered; screening extreme points with a principal curvature not exceeding a preset principal curvature threshold as first feature points according to a Gaussian difference pyramid of the second target image to be registered, and then screening the first feature points satisfying a preset condition in re-projection error and sampling probability as second feature points; solving a homogeneous linear equation set of pixel pairs matched with the second feature points by using an SVD method to obtain a transformation homography matrix, and then performing registration on the first target image to be registered to obtain a target image. The method breaks through the contradiction closed loop of "low efficiency and insufficient accuracy", and guarantees the safety of aviation operation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image registration method and an image registration apparatus. Background Technology

[0002] Airport runways, aprons, and taxiways are core areas for aircraft takeoff and landing. Foreign Object Debris (FOD) within these areas can cause serious safety incidents. Non-aircraft objects such as metal fragments, tools, gravel, plastic products, and bird carcasses can be ingested by aircraft engines or penetrate the fuselage, leading to mechanical failures or even crashes. Since aircraft accident rates are positively correlated with the number of FODs, real-time and accurate FOD detection is crucial for ensuring aviation safety. Currently, multi-technology solutions integrating millimeter-wave radar, optical imaging, and artificial intelligence (AI) algorithms are the mainstream approach for FOD detection. Image registration, as a core component of FOD detection, directly determines detection accuracy and robustness.

[0003] The problem with traditional image registration methods is that it is difficult to achieve both speed and accuracy. They cannot break through the contradiction of "low efficiency - insufficient accuracy", which makes it impossible to achieve FOD detection efficiently and with high accuracy, thus seriously restricting the effectiveness of airport security inspection. Summary of the Invention

[0004] This application provides an image registration method and device. By pre-aligning images using an optimal affine transformation model, the computational complexity and load of the image registration process are effectively reduced, thereby improving image registration efficiency. This shortens the single-frame processing time from minutes to seconds, meeting the real-time requirements of high-frequency detection in electronic equipment. Simultaneously, a three-dimensional extreme value search using a Gaussian difference pyramid combined with principal curvature constraints effectively filters edge noise points. Furthermore, an improved RANSAC algorithm is used to filter feature points under dual conditions (reprojection error and sampling probability), effectively reducing the mismatch rate under conditions of large camera viewing angle differences and rain, fog, and low light, thus improving image registration accuracy and reducing the risk of missed foreign object detection. In other words, this method overcomes the contradictory closed loop of "low efficiency - insufficient accuracy," enabling efficient and high-precision FOD detection, thereby improving airport security inspection efficiency and ensuring aviation operational safety.

[0005] This application provides an image registration method, including:

[0006] Acquire the target reference image and the first target registration image corresponding to the area to be inspected containing foreign object FOD;

[0007] Construct an optimal affine transformation model based on the target reference image and N preset pitch angles, where N is an integer greater than 1, and use the optimal affine transformation model to perform a linear transformation on the first target registration image to obtain a second target registration image;

[0008] Perform a three-dimensional extreme value search on the difference-of-Gaussians pyramid corresponding to the second target registration image, and screen out the extreme points with the principal curvature not exceeding the preset principal curvature threshold as the first feature points of the second target registration image; and use the improved random sample consensus (RANSAC) algorithm for all the first feature points to screen out the feature points whose reprojection error and sampling probability meet the preset conditions as the second feature points;

[0009] Solve the homogeneous linear equation system of the pixel point pairs matching the second feature points by using the singular value decomposition (SVD) method to obtain the transformation homography matrix; and register the first target registration image according to the transformation homography matrix to obtain the target image.

[0010] According to an image registration method provided by an embodiment of the present application, the constructing an optimal affine transformation model based on the target reference image and N preset pitch angles includes: obtaining the N preset pitch angles from within a preset pitch angle range; constructing N affine transformation models according to the N preset pitch angles and their respective corresponding optimal azimuth angles; performing feature point detection on the target reference image by using the N affine transformation models respectively to obtain the number of feature points and the spatial distribution entropy corresponding to each affine transformation model; determining n affine transformation models from the N affine transformation models in descending order of the number of feature points, where 0 < n < N; and taking the affine transformation model with the largest spatial distribution entropy among the n affine transformation models as the optimal affine transformation model.

[0011] According to an embodiment of this application, an image registration method is provided, wherein the Gaussian difference pyramid includes M Gaussian difference images, where M is an integer greater than 2, and the M Gaussian difference images include a target Gaussian difference image, which is a Gaussian difference image adjacent to two other Gaussian difference images; the step of performing a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and selecting extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature points of the second target image to be registered, includes: traversing the Gaussian difference pyramid in the target Gaussian difference image. For each candidate pixel, obtain the first pixel that shares a corner with the candidate pixel; from two Gaussian difference images adjacent to the target Gaussian difference image, determine the second pixel corresponding to the position of the candidate pixel and the first pixel; take the pixel with the largest or smallest brightness value among the candidate pixel, the first pixel and the second pixel as the extreme point; determine the principal curvature according to the Hessian matrix of the extreme point; from all extreme points, select the extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature point of the second target image to be registered.

[0012] According to an embodiment of this application, an image registration method is provided, wherein the improved Random Sample Consensus (RANSAC) algorithm is applied to all first feature points to select feature points whose reprojection error and sampling probability meet preset conditions as second feature points. This includes: randomly selecting P1 first sub-feature points and P2 second sub-feature points from all the first feature points, where P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of first feature points; calculating a homography matrix based on the P1 first sub-feature points; determining the reprojection error for each second sub-feature point based on the homography matrix, the pixel coordinates of the second sub-feature point, and the pixel coordinates of the corresponding pixel in the second target image to be registered; determining the sampling probability based on the reprojection error; and selecting second sub-feature points from the P2 second sub-feature points whose reprojection error is less than a preset error threshold and whose sampling probability is greater than a preset probability threshold as the second feature points.

[0013] According to an embodiment of this application, an image registration method is provided, wherein the homogeneous linear equation system of pixel pairs matching the second feature points is solved using the Singular Value Decomposition (SVD) method to obtain a transformation homography matrix. The method includes: determining pixels matching each second feature point from the target reference image based on the descriptors of all second feature points, and constructing pixel pairs; constructing homogeneous equations based on the pixel coordinates of each pixel pair; and solving the homogeneous linear equation system constructed from all homogeneous equations using the SVD method to obtain the transformation homography matrix.

[0014] According to an embodiment of this application, an image registration method is provided, wherein constructing N affine transformation models based on N preset pitch angles and their corresponding optimal azimuth angles includes: performing the following operations for each preset pitch angle: determining a tilt parameter based on the preset pitch angle, wherein the calculation formula for the tilt parameter is: , This represents the tilt parameter. The preset pitch angle is represented; the azimuth sampling interval is determined based on the tilt parameter, and the formula for calculating the azimuth sampling interval is: , This indicates the azimuth sampling interval. The constant is represented; the optimal azimuth angle is determined according to the azimuth angle sampling interval; and an affine transformation model is constructed according to the optimal azimuth angle and the preset elevation angle.

[0015] According to an embodiment of this application, an image registration method is provided, wherein determining the principal curvature based on the Hessian matrix of the extreme points includes: determining the principal curvature based on the extreme points in... The first and second partial derivatives in the direction, the extreme points at The Hessian matrix is ​​constructed using the second second-order partial derivative of the direction and the cross partial derivative of the first and second second-order partial derivatives; the principal curvature is determined based on the Hessian matrix, and the formula for calculating the principal curvature is as follows: ;in, Indicates the principal curvature; Represents the Hessian matrix; This represents the summation of the first second-order partial derivative and the second second-order partial derivative; This represents the difference between the product of the first second-order partial derivative and the second second-order partial derivative and the square of the cross partial derivative.

[0016] According to an embodiment of this application, an image registration method is provided, wherein the homogeneous linear equation system is: , Represents the coefficient matrix. The transformed homography matrix is ​​represented by a flattened vector constructed from all elements of the transformed homography matrix. The step of solving the homogeneous linear equation system constructed from all homogeneous equations using the SVD method to obtain the transformed homography matrix includes: using the SVD method to flatten the coefficient matrix... Decomposition yields the right singular vector matrix. ; the right singular vector matrix The last column element in the vector is used as the flattened vector. According to the flattened vector All elements in the matrix are used to construct the transformation homography matrix.

[0017] According to an embodiment of this application, an image registration method is provided, wherein obtaining a target reference image and a first target registration image corresponding to an inspection area containing foreign object (FOD) includes: acquiring images of the inspection area captured by a multi-camera array; using an image captured by one camera in the multi-camera array as a first reference image and images captured by the remaining cameras as first registration images; scaling the first reference image and the first registration image respectively using a preset sampling operator to obtain a second reference image and a second registration image; and performing high-frequency noise removal on the second reference image and the second registration image respectively using a Gaussian filter to obtain the target reference image and the first target registration image.

[0018] This application also provides an image registration device, including:

[0019] The acquisition module is used to acquire the target reference image and the first target registration image corresponding to the area to be inspected containing foreign object FOD.

[0020] The transformation module is used to construct an optimal affine transformation model based on the target reference image and N preset pitch angles, where N is an integer greater than 1, and to perform a linear transformation on the first target image to be registered using the optimal affine transformation model to obtain the second target image to be registered.

[0021] The filtering module is used to perform a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and to filter extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature points of the second target image to be registered; and to use an improved Random Sample Consensus (RANSAC) algorithm on all the first feature points to filter feature points whose reprojection error and sampling probability meet preset conditions as the second feature points.

[0022] The registration module is used to solve the homogeneous linear equations of the pixel pairs that match the second feature point using the singular value decomposition (SVD) method to obtain the transformation homography matrix; and to register the first target image to be registered according to the transformation homography matrix to obtain the target image.

[0023] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image registration method as described above.

[0024] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image registration method as described above.

[0025] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the image registration method as described above.

[0026] The image registration method and image registration device provided in this application acquire a target reference image and a first target image to be registered corresponding to the area to be inspected containing foreign object (FOD). Based on the target reference image and N preset pitch angles, an optimal affine transformation model is constructed, where N is an integer greater than 1. The optimal affine transformation model is then used to perform a linear transformation on the first target image to be registered, resulting in a second target image to be registered. A three-dimensional extreme value search is performed on the difference-of-Gaussian pyramid corresponding to the second target image to be registered, and extreme points whose principal curvature does not exceed a preset principal curvature threshold are selected as first feature points of the second target image to be registered. An improved Random Sample Consensus (RANSAC) algorithm is applied to all first feature points, and feature points whose reprojection error and sampling probability satisfy preset conditions are selected as second feature points. The singular value decomposition (SVD) method is used to solve the homogeneous linear equations of pixel pairs matching the second feature points to obtain a transformation homography matrix. Finally, the first target image to be registered is registered based on the transformation homography matrix to obtain the target image. This method pre-aligns images using an optimal affine transformation model, effectively reducing the computational complexity and load of the image registration process, thereby improving registration efficiency and shortening single-frame processing time from minutes to seconds, meeting the real-time requirements of high-frequency electronic equipment detection. Simultaneously, it employs a three-dimensional extremum search using a Gaussian difference pyramid combined with principal curvature constraints to effectively filter edge noise points. Furthermore, an improved RANSAC algorithm uses dual conditions (reprojection error and sampling probability) to filter feature points, effectively reducing mismatch rates under conditions of large camera viewing angle differences and rain, fog, or low light, thus improving image registration accuracy and reducing the risk of missed foreign object detection. In other words, this method overcomes the contradictory loop of "low efficiency - insufficient accuracy," enabling efficient and high-precision FOD detection, thereby improving airport security inspection efficiency and ensuring aviation operational safety. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the arrangement of a quad-camera array provided in an embodiment of this application;

[0029] Figure 2This is a schematic diagram of a scene where an image is transmitted between a quad-camera array and an electronic device, as provided in an embodiment of this application.

[0030] Figure 3 This is a schematic flowchart of the image registration method provided in the embodiments of this application;

[0031] Figure 4 This is a schematic diagram of the image registration device provided in the embodiments of this application;

[0032] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] To better understand the embodiments of this application, the background technology is described in detail below:

[0035] Traditional image registration methods suffer from the problem of not being able to achieve both speed and accuracy. Specifically, while traditional image registration methods (such as feature matching-based methods) are somewhat adaptable to large changes in viewing angle, their computational load is too high, resulting in low image registration efficiency. This leads to poor real-time performance in foreign object detection, with single-frame registration taking a long time (on the order of minutes), which cannot meet the high-frequency detection requirements of electronic equipment (such as airport vehicle-mounted inspection systems). On the other hand, optimized algorithms show a significant increase in mismatch rate in complex pavement environments, especially under conditions of large camera viewing angle differences and rain, fog, and low light, resulting in insufficient image registration accuracy and thus the risk of missing foreign objects.

[0036] In summary, traditional image registration methods cannot break through the contradictory loop of "low efficiency and insufficient accuracy," resulting in the inability to achieve FOD detection efficiently and with high accuracy, which in turn severely restricts the effectiveness of airport security inspections.

[0037] To address the aforementioned technical problems, this application provides an image registration method. The method involves acquiring a target reference image and a first target image to be registered, corresponding to the area to be inspected containing foreign object (FOD). Based on the target reference image and N preset pitch angles, an optimal affine transformation model is constructed, where N is an integer greater than 1. This optimal affine transformation model is then used to perform a linear transformation on the first target image to be registered, resulting in a second target image to be registered. A three-dimensional extreme value search is performed on the difference-of-Gaussian pyramid corresponding to the second target image to be registered, selecting extreme points whose principal curvature does not exceed a preset principal curvature threshold as first feature points of the second target image to be registered. An improved Random Sample Consensus (RANSAC) algorithm is applied to all first feature points, selecting feature points whose reprojection error and sampling probability satisfy preset conditions as second feature points. The singular value decomposition (SVD) method is used to solve the homogeneous linear equations of pixel pairs matching the second feature points, obtaining a transformation homography matrix. Finally, the first target image to be registered is registered based on the transformation homography matrix to obtain the target image. This method pre-aligns images using an optimal affine transformation model, effectively reducing the computational complexity and load of the image registration process, thereby improving registration efficiency and shortening single-frame processing time from minutes to seconds, meeting the real-time requirements of high-frequency electronic equipment detection. Simultaneously, it employs a three-dimensional extremum search using a Gaussian difference pyramid combined with principal curvature constraints to effectively filter edge noise points. Furthermore, an improved RANSAC algorithm uses dual conditions (reprojection error and sampling probability) to filter feature points, effectively reducing mismatch rates under conditions of large camera viewing angle differences and rain, fog, or low light, thus improving image registration accuracy and reducing the risk of missed foreign object detection. In other words, this method overcomes the contradictory loop of "low efficiency - insufficient accuracy," enabling efficient and high-precision FOD detection, thereby improving airport security inspection efficiency and ensuring aviation operational safety.

[0038] To better understand the embodiments of this application, the multi-view vision system containing the multi-view camera array involved in the embodiments of this application will be described in detail below:

[0039] I. Hardware Architecture of Multi-View Vision System

[0040] The hardware architecture of a multi-view vision system mainly consists of an optical sensing unit, an ambient light control unit, and a trigger control component. The optical sensing unit employs a multi-view camera array and a supporting mechanical structure to acquire visual information, i.e., image capture. The ambient light control unit uses an adjustable light source to achieve illumination compensation. The trigger control component is equipped with programmable timing signals to enable synchronous image acquisition from the multi-view cameras.

[0041] Optionally, the core functions of the above multi-view vision system may include: real-time synchronous capture of multiple optical signals based on a hardware synchronization mechanism, optimization of environmental light conditions using active lighting technology to improve image acquisition adaptability, and timing control for multi-device collaborative work through an external trigger interface, etc.

[0042] It should be noted that the above optical sensing unit usually adopts a global shutter Complementary Metal-Oxide-Semiconductor (CMOS) sensor array to construct a multi-view imaging system (i.e., a multi-view camera array). Optionally, the cameras in the multi-view camera array are all high-speed industrial cameras, which can obtain rich information of the area to be inspected through multi-view images for subsequent image registration. In addition, the gimbal cloud platform bracket supporting the above optical sensing unit enables precise adjustment of multiple degrees of freedom to meet the attitude calibration requirements in complex installation scenarios.

[0043] Exemplarily, in the case where the multi-view camera array is a four-view camera array, the arrangement of the four-view camera array is as Figure 1 shown. As can be seen from Figure 1 , the four-view camera array includes four cameras, namely camera C1, camera C2, camera C3, and camera C4. These four cameras are all arranged diagonally at a preset angle to photograph the area to be inspected, and can maximize the stereoscopic feature expression of foreign object debris (FOD) while ensuring field of view coverage.

[0044] Combined with Figure 1 , as Figure 2 shown, it is a schematic diagram of the scenario for image transmission between the four-view camera array provided by an embodiment of the present application and an electronic device. As can be seen from Figure 2 , after the above four-view camera array performs image acquisition on the area to be inspected, all the acquired images can be transmitted to the electronic device through a switch using a 10 Gigabit Ethernet cable. The electronic device then receives all the images through a switch using a 10 Gigabit Ethernet cable for subsequent image registration.

[0045] II. Software Architecture of the Multi-View Vision System

[0046] The software architecture of the multi-view vision system is developed based on Python 3.8.19, and the core functions are implemented by key libraries. Specifically, OpenCV 3.4.10.35 is responsible for underlying image acquisition, processing, and real-time operations; Scikit-image 0.21.0 provides advanced image analysis algorithms (such as segmentation, morphological processing, etc.); Scikit-learn 1.3.2 supports machine learning tasks; Numpy 1.24.4 is used as the core data carrier to achieve efficient data exchange between libraries. The multi-view vision system uses Joblib 1.3.2 to achieve parallel computing acceleration.

[0047] The user interaction layer is developed using the PyQt 55.15.10 framework, with a graphical user interface (GUI) built using Qt Designer to enable parameter control, data input, and result visualization. The developed software runs on Windows 10, utilizing an i7-12700K processor and 64GB of memory, providing efficient and stable image processing and intelligent analysis capabilities for semi-miniature dual-row systems in electronic devices.

[0048] The execution subject involved in the embodiments of this application will be described in detail below:

[0049] The execution entity involved in the embodiments of this application can be an image registration device or an electronic device. Optionally, the electronic device may include: a control terminal, an airport vehicle-mounted detection system, a computer, a mobile terminal, and a wearable device, etc.

[0050] The image registration method provided in this application embodiment will be described in detail below using an electronic device as an example:

[0051] Figure 3 This is a schematic flowchart of the image registration method provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps 301-304.

[0052] Step 301: Obtain the target reference image and the first target registration image corresponding to the area to be inspected containing foreign object FOD.

[0053] The aforementioned target reference image serves as a reference frame for registration, used to adjust the spatial alignment of the first target image to be registered.

[0054] The aforementioned first target image to be registered refers to the image to be registered that corresponds to the aforementioned target reference image and has undergone the same preprocessing. There is at least one such first target image to be registered.

[0055] It should be noted that the timing of the electronic device acquiring the target reference image and the first target image to be registered is not limited.

[0056] The following section elaborates on how electronic devices acquire the target reference image and the first target registration image corresponding to the area under inspection containing foreign object (FOD):

[0057] In some embodiments, the electronic device acquiring a target reference image and a first target registration image corresponding to an inspection area containing foreign object (FOD) may include: the electronic device acquiring images of the inspection area captured by a multi-camera array; the electronic device using an image captured by one camera in the multi-camera array as the first reference image and the images captured by the remaining cameras as the first registration images; the electronic device scaling the first reference image and the first registration image using a preset sampling operator to obtain a second reference image and a second registration image; and the electronic device performing high-frequency noise cancellation on the second reference image and the second registration image using a Gaussian filter to obtain the target reference image and the first target registration image.

[0058] The aforementioned preset sampling operator is an image scaling algorithm that generates thumbnails by reducing image resolution, thereby reducing the computational load of subsequent image registration processing while preserving key features.

[0059] The Gaussian filter mentioned above is a linear smoothing filter. It is based on the weighted smoothing operation of the Gaussian kernel. It uses a Gaussian function to perform a weighted average of image pixels, which attenuates high-frequency noise such as jagged edges and noise generated during scaling, while improving the robustness of the image registration process.

[0060] In this embodiment, after the multi-camera array acquires images of the area to be inspected, it can send all the acquired images to an electronic device. Upon receiving all the images, the electronic device can use the image acquired by any one camera in the multi-camera array as the first reference image, and the images acquired by the remaining cameras in the multi-camera array as the first images to be registered. It is understood that, combined with... Figure 1 If the electronic device uses the image captured by camera C1 as the first reference image, it will use the images captured by camera C2, camera C3 and camera C4 as the first images to be registered; if the electronic device uses the image captured by camera C2 as the first reference image, it will use the images captured by camera C1, camera C3 and camera C4 as the first images to be registered, without specific limitations here.

[0061] Then, the electronic device downsamples the first reference image and the first image to be registered using a combination of quadratic sampling and Gaussian filtering. That is, the downsampling refers to reducing the image size by a fixed ratio, while using a Gaussian filter to eliminate noise that appears during scaling. The specific process is as follows:

[0062] During the downsampling of the first reference image, the electronic device can scale the first reference image using a preset sampling operator to obtain a second reference image. The calculation formula for the second reference image is as follows: ,in, Indicates the second reference image, This indicates the preset sampling operator. The first reference image is represented; further, the electronic device performs high-frequency noise removal on the second reference image using a Gaussian filter to obtain the target reference image, which is calculated using the following formula: ,in, Represents the target reference image. This represents a Gaussian filter.

[0063] During the downsampling process of the first image to be registered, the electronic device can scale the first image to be registered using the aforementioned preset sampling operator to obtain the second image to be registered. The calculation formula for the second image to be registered is as follows: ,in, This represents the second image to be registered. This represents the first image to be registered; further, the electronic device performs high-frequency noise removal on the second image to be registered using the aforementioned Gaussian filter to obtain the target image to be registered. The calculation formula for the target image to be registered is as follows: ,in, This represents the target image to be registered.

[0064] For example, assuming the above-mentioned preset sampling operator Sampling factor The sampling operator, the above Gaussian filter A Gaussian filter with a standard deviation of 1.5 is used. Taking an 800×800 pixel first reference image as an example, the resolution of the target reference image obtained after downsampling is reduced to 267×267 pixels. The data volume of the target reference image is reduced to 1 / 9 of that of the first reference image. The Gaussian filter uses a 9×9 kernel function to eliminate high-frequency noise caused by pixel resampling during downsampling, and retains structural features larger than 10 pixels in the target reference image (corresponding to foreign objects larger than 5mm in an airport runway), laying the foundation for the subsequent image registration process.

[0065] It should be noted that electronic devices, through downsampling-filtering combined preprocessing technology, can compress the image data to 1 / 9 of the original image while retaining the key features of foreign objects larger than 5mm, thus providing engineering feasibility for embedded device deployment.

[0066] Step 302: Based on the target reference image and N preset pitch angles, construct the optimal affine transformation model, and use the optimal affine transformation model to perform a linear transformation on the first target image to be registered, to obtain the second target image to be registered.

[0067] Where N is an integer greater than 1.

[0068] The preset pitch angles described above describe the camera lens's tilt or downward position relative to the horizontal direction. Specifically, they refer to the camera's tilt angle in the vertical plane (i.e., the front-to-back direction), usually with the horizontal plane as the reference (0°), upward being a positive angle and downward being a negative angle. It should be noted that any two preset pitch angles among the N preset pitch angles are different.

[0069] Affine transformation is a linear transformation model that describes the geometric deformation of a two-dimensional plane. By simulating camera pitch, rotation, and other attitude changes, it maps images from different viewpoints to a unified geometric space. This affine transformation model includes operations such as translation, rotation, scaling, and shearing to achieve attitude normalization of multi-view images. Specifically, the optimal affine transformation model described above achieves the best alignment between the target reference image and the first target image to be registered.

[0070] In step 302, the electronic device employs a feature matching algorithm extended from SIFT, namely the Affine Scale-Invariant Feature Transform (ASIFT) algorithm, to address the image distortion problem caused by large viewing angle differences. This ASIFT algorithm simulates the imaging process of a camera at different preset pitch angles and optimal azimuth angles, generating multiple sets of affine-transformed image sequences, and then extracting affine-invariant feature points from them. Specifically, the electronic device constructs N affine transformation models based on the target reference image and N preset pitch angles, selects the optimal affine transformation model from these N models, and then uses this optimal affine transformation model to perform a linear transformation on the first target image to be registered, obtaining the second target image to be registered with the best transformation effect, for subsequent accurate image registration. The entire process proposes a collaborative sampling mechanism that combines a preset pitch angle constraint range with an optimal azimuth angle dynamic interval strategy. By compressing the parameter space, redundant calculations are significantly reduced, effectively lowering the computational complexity of the image registration process and reducing the computational load, thereby improving image registration efficiency. This reduces the processing time per frame from minutes to seconds, meeting the real-time requirements of high-frequency detection in electronic devices.

[0071] The following section elaborates on how electronic equipment constructs the optimal affine transformation model based on a target reference image and N preset pitch angles:

[0072] In some embodiments, an electronic device constructs an optimal affine transformation model according to a target reference image and N preset pitch angles, which may include: the electronic device obtains N preset pitch angles from within a preset pitch angle range; the electronic device constructs N affine transformation models according to the N preset pitch angles and their respective corresponding optimal azimuth angles; the electronic device performs feature point detection on the target reference image using the N affine transformation models respectively, and obtains the number of feature points and the spatial distribution entropy corresponding to each affine transformation model; the electronic device determines n affine transformation models from the N affine transformation models in descending order of the number of feature points, where 0 < n < N; and takes the affine transformation model with the largest spatial distribution entropy among the n affine transformation models as the optimal affine transformation model.

[0073] Among them, the preset pitch angle range is used to limit the pitch movement range of the multi-camera array during actual operation, ensuring that all possible perspectives of the area to be inspected are covered. Specifically, it refers to the tilt angle interval allowed for the multi-camera array in the vertical plane. Usually, with the horizontal direction as the reference, positive and negative boundary values are set respectively for upward (raising the head) and downward (lowering the head).

[0074] The azimuth angle refers to the rotation angle of the camera in the horizontal plane (taking due north or the reference direction as 0°, increasing clockwise). And the above optimal azimuth angle is the best horizontal rotation angle obtained through calibration or optimization for a certain preset pitch angle, making the matching degree between the camera view and the target reference image the highest.

[0075] The above spatial distribution entropy is an index for measuring the uniformity of feature point distribution in an image. Among them, the larger the spatial distribution entropy, the more uniform the feature point distribution is characterized. In addition, the introduction of this spatial distribution entropy effectively reduces the interference of local noise or abnormal feature points on the selection of the optimal affine transformation model.

[0076] In this embodiment, addressing the issue of camera perspective differences in a multi-camera array, since a preset pitch angle range (e.g., 45°, 70°) can cover a significant portion of the actual shooting angles, the electronic device can limit N preset pitch angles within this range for affine transformation simulation. Based on this, the electronic device can first randomly select N preset pitch angles from the preset pitch angle range and determine the optimal azimuth angle corresponding to each of these N preset pitch angles. In this way, a preset pitch angle and its corresponding optimal azimuth angle can jointly determine a spatial attitude (pitch + horizontal rotation) of the camera. This spatial attitude is an affine transformation model. Thus, the electronic device can ultimately determine N affine transformation models. The entire process, through the combination of the preset pitch angle range and the optimal azimuth angle, covers various perspective changes that may exist due to foreign objects, avoiding the potential problems caused by a single affine transformation model. The electronic device first performs feature point detection on the target reference image using N affine transformation models, then counts the number of feature points and spatial distribution entropy corresponding to each affine transformation model. This allows the electronic device to determine the number of N feature points and N spatial distribution entropies. Next, the electronic device sorts the N feature point counts in descending order and selects the top n feature point counts. This initial screening of the top n affine transformation models with the most feature points ensures sufficient matching points to support the transformation calculation, avoiding registration failure due to insufficient features. Finally, the electronic device determines the maximum spatial distribution entropy from the spatial distribution entropies corresponding to the top n affine transformation models and uses the affine transformation model with the maximum spatial distribution entropy as the optimal affine transformation model. This process, based on a sufficient number of feature points, selects the affine transformation model with the most evenly distributed feature points as the optimal affine transformation model, preventing feature points from concentrating in local areas (such as textured areas) and thus improving global alignment accuracy.

[0077] It should be noted that, in response to the challenge of large-angle registration for electronic devices, a downsampling constraint-based parameter compression technique is adopted in the feature point matching stage. Specifically, the computational load is reduced by 3 times image thumbnail preprocessing, and the effective range of the camera's preset pitch angle is locked from 45° to 70°. This breaks through the traditional full-angle redundant sampling. Combined with the azimuth angle dynamic interval strategy, the number of affine transformation simulations is sharply reduced from 61 times in the traditional full-angle sampling to 24 times. Combined with 3 times image thumbnail preprocessing, the registration time per frame is reduced, and the computational efficiency is greatly improved by 5 times, which fully meets the real-time detection requirements of electronic devices.

[0078] In some embodiments, the electronic device constructs N affine transformation models based on N preset pitch angles and their corresponding optimal azimuth angles. This may include performing the following operations for each preset pitch angle: The electronic device determines a tilt parameter based on the preset pitch angle, and the calculation formula for the tilt parameter is: , Indicates the tilt parameter. This indicates the preset pitch angle; the electronic device determines the azimuth sampling interval based on the tilt parameters, and the formula for calculating the azimuth sampling interval is: , Indicates the azimuth sampling interval. The constant represents the azimuth angle; the electronic device determines the optimal azimuth angle based on the azimuth angle sampling interval; the electronic device constructs an affine transformation model based on the optimal azimuth angle and the preset elevation angle.

[0079] The tilt parameter mentioned above is a scale factor used to quantify the influence of the preset pitch angle on the geometric deformation of the target reference image.

[0080] The aforementioned azimuth sampling interval is the angle difference between adjacent sampled azimuth angles in the horizontal plane.

[0081] Since the determination process for each affine transformation model is identical, the electronic device can perform the following operations for each preset pitch angle using a parameterized sampling strategy: The sampling density in the horizontal direction (azimuth angle) needs to be determined based on the tilt parameters. To maintain the uniformity of feature matching, the electronic device can use a preset pitch angle to calculate the tilt parameter. Furthermore, a larger tilt parameter (i.e., a larger preset pitch angle) will result in a larger azimuth sampling interval. The smaller the value, the denser the sampling in the horizontal direction, to compensate for feature sparsity caused by pitch distortion. A constant can be introduced. =72°, used to balance sampling density and computational efficiency. Based on this, the electronic device substitutes the above tilt parameter into the calculation formula for the azimuth sampling interval to obtain the azimuth sampling interval; then, the electronic device samples the range (0°, 360°) according to the azimuth sampling interval to obtain multiple azimuth angles, and determines the optimal azimuth angle from these multiple azimuth angles. Then, by combining the aforementioned preset pitch angles, an affine transformation model is constructed. In this way, the electronic device can ultimately determine N affine transformation models.

[0082] For example, assuming N is 2, then the two affine transformation models are affine transformation model 1 and affine transformation model 2, where affine transformation model 1 is ( =45°, =45°), affine transformation model 2 is ( =60°, =36°).

[0083] Step 303: Perform a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and select extreme points whose principal curvature does not exceed the preset principal curvature threshold as the first feature points of the second target image to be registered; and use the improved Random Sample Consistency (RANSAC) algorithm on all first feature points to select feature points whose reprojection error and sampling probability meet the preset conditions as second feature points.

[0084] The aforementioned Difference of Gaussians (DoG) pyramid is a multi-scale image representation constructed by subtracting Gaussian blurred versions of the same image (i.e., the second target image to be registered) at different scales.

[0085] The principal curvature mentioned above is an index describing the maximum and minimum curvature of a surface at a certain extreme point. It is used to describe the geometric characteristics of the local surface (or image intensity surface) where the key point (extreme point) is located. Its main purpose is to filter out points with strong edge response (such as edge points) and retain stable corner points or spot features.

[0086] The aforementioned reprojection error is a key indicator used to evaluate the quality of feature point matching and the accuracy of geometric models.

[0087] Sampling probability refers to the probability weight of each feature point being selected to participate in model estimation in the improved Random Sample Consensus (RANSAC) algorithm. RANSAC is a robust model fitting method that uses iterative sampling to select the optimal model from noisy data.

[0088] In step 303, after determining the Gaussian difference pyramid corresponding to the second target image to be registered, the electronic device can perform a three-dimensional extreme value search on the Gaussian difference pyramid. Based on the principal curvature, the extreme points are initially screened to remove edge-sensitive points and retain robust features, resulting in multiple first feature points of better quality. Then, the electronic device uses an improved RANSAC algorithm to perform a second screening on all first feature points based on reprojection error and sampling probability, avoiding the blind randomness of traditional RANSAC and accelerating convergence, resulting in multiple second feature points of better accuracy for subsequent accurate image registration.

[0089] Optionally, before step 303, the method may further include: the electronic device using a Gaussian kernel with M+1 standard deviations to blur the second target image to be registered, respectively, to obtain M+1 Gaussian images, where M is an integer greater than or equal to 3; the electronic device subtracting two adjacent Gaussian images in the M+1 Gaussian images to construct a Gaussian difference pyramid, which includes M Gaussian difference images.

[0090] For example, if M=4, then the standard deviation The quantity is 5, respectively =1.0、 =1.4、 =2.0、 =2.4 and =4.8. Based on this, the electronic device can use these 5 standard deviations of Gaussian kernels to blur the second target image to be registered, generating 5 Gaussian images with different degrees of blurring. This can highlight those features that are obvious under different scaling degrees. Then, the electronic device subtracts the two adjacent Gaussian images in the above 5 Gaussian images to obtain 4 Gaussian difference images to construct a Gaussian difference pyramid.

[0091] With the first one standard deviation and the +1 standard deviation For example, > , ={1,…,M}, electronic devices use standard deviation The Gaussian kernel is used to blur the second target image to be registered, generating a Gaussian image α, and the standard deviation is used. The Gaussian kernel blurs the second target image to be registered, generating a Gaussian image β, where Gaussian image β and the aforementioned Gaussian image α are two adjacent Gaussian images. Then, the electronic device subtracts Gaussian image β from Gaussian image α to obtain a difference-of-Gaussian image. The calculation formula for the difference-of-Gaussian image is as follows: .

[0092] in, Represents the difference-of-Gaussian image, ( () represents the pixel coordinates at the corresponding position; β represents the Gaussian image. = , Standard deviation Gaussian kernel, This represents the image to be registered for the second target. Indicates the convolution operation; Let α represent the Gaussian image. = , Standard deviation Gaussian kernel.

[0093] The following section elaborates on how the electronic device performs a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, selecting extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature points of the second target image to be registered:

[0094] In some embodiments, the Gaussian difference pyramid includes M Gaussian difference images, where M is an integer greater than 2, and the M Gaussian difference images include a target Gaussian difference image, which is a Gaussian difference image that is adjacent to two Gaussian difference images.

[0095] Based on this, the electronic device performs a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and selects extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature points of the second target image to be registered. This can include: traversing each candidate pixel in the target Gaussian difference image, the electronic device obtaining the first pixel that shares a corner with the candidate pixel; determining the second pixel corresponding to the position of the candidate pixel and the first pixel from two Gaussian difference images adjacent to the target Gaussian difference image; selecting the pixel with the largest or smallest brightness value among the candidate pixel, the first pixel, and the second pixel as the extreme point; determining the principal curvature according to the Hessian matrix of the extreme point; and selecting extreme points whose principal curvature does not exceed a preset principal curvature threshold from all extreme points as the first feature points of the second target image to be registered.

[0096] The Hessian matrix, also known as the Hessian matrix, is a square matrix composed of the second-order partial derivatives of a multivariable function. In image processing, it is used to describe the curvature characteristics of local structures.

[0097] In the initial screening process, classic feature extraction algorithms (such as Scale-Invariant Feature Transform (SIFT)) can be used. The core idea is to use the Difference of Gaussian (DG) pyramid to detect extreme points in scale space as candidate pixels, also known as candidate feature points, ensuring that the final determined first feature point possesses scale, rotation, and illumination invariance. Specifically, the electronic device first determines the DG pyramid corresponding to the second target image to be registered, and then determines the target DG image from the M DG images included in the DG pyramid. Next, the electronic device traverses each candidate pixel in the target DG image, obtaining the first pixel sharing a corner with that candidate pixel. It can be understood that if the candidate pixel is a corner, the number of first pixels is 3; if the candidate pixel is an edge, the number of first pixels is 5; and if the candidate pixel is neither a corner nor an edge, the number of first pixels is 8. Then, the electronic device extracts the first pixel from the M DG images included in the DG pyramid... The electronic device identifies two Gaussian difference images adjacent to the target Gaussian difference image in the image, and determines the second pixel corresponding to the candidate pixel and the first pixel from these two Gaussian difference images. Subsequently, the electronic device selects the pixel with the largest (brightest) or smallest (darkest) brightness value among the candidate pixel, the first pixel, and the second pixel as the extreme point, and calculates the Hessian matrix of the extreme point to determine the principal curvature of the extreme point. Finally, the electronic device selects extreme points from all extreme points that are not very sensitive to changes in viewing angle, that is, extreme points whose principal curvature does not exceed a preset principal curvature threshold, as the first feature points of the second target image to be registered.

[0098] For example, assuming the candidate pixel is a corner point, the surrounding pixels of the corner point are 2×2 pixels. These 2×2 pixels include the corner point and the first pixel sharing the corner point. In this case, the electronic device determines the second pixel corresponding to the position of the 2×2 pixels from two adjacent Gaussian difference images, and then takes the brightest or darkest pixel in the 2×2×3 range as the extreme point. Alternatively, assuming the candidate pixel is an edge point, the surrounding pixels of the edge point are 2×3 pixels. These 2×3 pixels include the edge point and the first pixel sharing the corner point. In this case, the electronic device determines the second pixel corresponding to the position of the 2×2 pixels from two adjacent Gaussian difference images, and then takes the brightest or darkest pixel in the 2×2×3 range as the extreme point. The second pixel corresponding to the position of 2×3 pixels is determined from two adjacent Gaussian difference images. Then, the brightest or darkest pixel in the 2×3×3 range is taken as the extreme point. Assuming that the above candidate pixel is neither a corner point nor an edge point, the surrounding pixels of the candidate pixel are 3×3 pixels. These 3×3 pixels include the candidate pixel and the first pixel that shares a corner point with the candidate pixel. At this time, the electronic device determines the second pixel corresponding to the position of 3×3 pixels from two adjacent Gaussian difference images. Then, the brightest or darkest pixel in the 3×3×3 range is taken as the extreme point.

[0099] Optionally, the above Hessian matrix is .

[0100] in, Indicates the extreme point at The first and second partial derivatives in the direction; Indicates the extreme point at The second partial derivative in the direction, This represents the cross-partial derivative of the first and second order partial derivatives and the second and second order partial derivatives.

[0101] In some embodiments, the electronic device determines the principal curvature based on the Hessian matrix of the extreme points, which may include: the electronic device determining the principal curvature based on the extreme points at... The first and second partial derivatives in the direction, and the extreme points are at The Hessian matrix is ​​constructed using the second and second-order partial derivatives of the directions, and the cross-partial derivatives of the first and second-order partial derivatives. Based on the Hessian matrix, the electronic device determines the principal curvature, and the formula for calculating the principal curvature is: .

[0102] in, Indicates the principal curvature; Represents the Hessian matrix; This represents the summation of the first and second-order partial derivatives, i.e. ; This represents the difference between the product of the first and second-order partial derivatives and the square of the cross partial derivatives, i.e. .

[0103] For example, suppose the preset principal curvature threshold is set to 100, if the principal curvature If the value is >100, it indicates that the extreme point has a large difference in principal curvature in the two directions. This extreme point is sensitive to changes in viewing angle and is not the edge point of a foreign object. In this case, the extreme point should be removed. If the principal curvature is >100, it means that the extreme point has a large difference in principal curvature in the two directions. This extreme point is sensitive to changes in viewing angle and is not the edge point of a foreign object. If the value is 100, meaning it does not exceed 100, it indicates that the difference in principal curvature between the two directions is small. This extreme point is not very sensitive to changes in viewing angle and is the edge point of foreign objects. In this case, the extreme point can be retained.

[0104] The following section elaborates on how electronic devices use an improved Random Sample Consensus (RANSAC) algorithm to select feature points whose reprojection error and sampling probability meet preset conditions as second feature points for all first feature points:

[0105] In some embodiments, the electronic device employs an improved Random Sample Consensus (RANSAC) algorithm on all first feature points to select feature points whose reprojection error and sampling probability meet preset conditions as second feature points. This may include: the electronic device randomly selecting P1 first sub-feature points and P2 second sub-feature points from all first feature points, where P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of first feature points; the electronic device calculates a homography matrix based on the P1 first sub-feature points; for each second sub-feature point, the electronic device determines the reprojection error based on the homography matrix, the pixel coordinates of the second sub-feature point, and the pixel coordinates of the corresponding pixel in the second target image to be registered; and determines the sampling probability based on the reprojection error; the electronic device selects second sub-feature points from the P2 second sub-feature points whose reprojection error is less than a preset error threshold and whose sampling probability is greater than a preset probability threshold as second feature points.

[0106] The homography matrix mentioned above is a 3×3 homogeneous matrix, a mapping matrix used to describe the perspective transformation relationship between two planar images, and can be used... express.

[0107] and Indicates the camera's scaling factor; and This indicates the shearing transformation that affects the image; and Indicates the translation component; and Indicates perspective distortion correction; This represents the homogeneous coordinate normalization factor, which is usually set to 1.

[0108] During the secondary screening process, an improved RANSAC algorithm can be used to screen all first feature points. Specifically, the electronic device in the first... In the first iteration, P1 first sub-feature points and P2 second sub-feature points are randomly selected, and the homography matrix corresponding to these P1 first sub-feature points is determined. Typically, P1 is set to 4. Then, for the P2 second sub-feature points... The second sub-feature point is used to verify geometric consistency. Specifically, the electronic device verifies geometric consistency based on the homography matrix and the second sub-feature point. Pixel coordinates of the second sub-feature point and the second target image to be registered with the first The pixel coordinates of the second sub-feature points correspond to the pixel points. Determine the first The reprojection error of the second sub-feature point; then, the electronic device determines the first based on the reprojection error. The initial sampling probability of each of the P2 second sub-feature points is obtained, and then the sampling probability is calculated. Based on this, the electronic device can finally determine the reprojection error and sampling probability of each of the P2 second sub-feature points. Finally, the electronic device selects the second sub-feature points from all the first feature points whose reprojection error is less than a preset error threshold (e.g., 1.5) and whose sampling probability is greater than a preset probability threshold (e.g., 0.9) as the second feature points.

[0109] The formula for calculating this reprojection error is as follows: , This represents the L2 norm, also known as the Euclidean norm.

[0110] The formula for calculating the initial sampling probability is: .

[0111] in, Indicates the first The second sub-feature point in the th... The weights for the next iteration; Indicates the first The second sub-feature point in the th... The weights for the next iteration; This represents the weight decay factor based on a Gaussian kernel, exemplarily using... Iterate the above weights 10 times.

[0112] The formula for calculating the sampling probability is: .

[0113] Indicates the first The sampling probability of the second sub-feature point; This represents the sum of sampling probabilities for all second sub-feature points.

[0114] It should be noted that, in response to the challenge of large-viewpoint registration of electronic devices, a dual verification mechanism was developed in the mismatch suppression stage. Specifically, a hypothesis filtering process of 4-point modeling + 1-point pre-detection in a 5-point sample set is adopted. In each RANSAC iteration, the geometric consistency is verified by the 5th sample point, which quickly screens out erroneous models. Combined with dynamic weight adjustment based on residuals, the mismatch interference in complex environments is significantly reduced, and high-precision registration is finally achieved to meet the real-time detection requirements of airport vehicle-mounted systems. At the same time, it significantly suppresses interference from complex pavement textures, and the mismatch rate remains stable even under large viewing angle differences (>60°) and rain, fog, and low light conditions. Compared with traditional image registration methods, it reduces the risk of missing foreign objects.

[0115] Optionally, during the process of filtering all first feature points using the improved RANSAC algorithm, the electronic device can calculate the spatial dispersion of the sampling point set (i.e., all first feature points); if the spatial dispersion is less than a preset dispersion threshold (e.g., 0.3), resampling is triggered to obtain new second feature points, ensuring that the point set is evenly distributed.

[0116] The spatial dispersion mentioned above is used to measure the uniformity of the distribution of the sampling point set. The larger the spatial dispersion, the more uneven the distribution of the sampling point set (it may be concentrated in a local area or there may be outliers). The closer the spatial dispersion is to 0, the more uniform the distribution of the sampling point set (that is, the maximum and minimum values ​​are close to the mean).

[0117] Optionally, the formula for calculating the spatial dispersion is: .

[0118] This represents the x-coordinate components of all points in the sampling point set; Represents the ordinate components of all points in the sampling point set; This represents the maximum value among all vertical and horizontal coordinate components. This represents the minimum value among all vertical and horizontal coordinate components. This represents the arithmetic mean of all vertical and horizontal axis components.

[0119] It should be noted that the dynamic probability update method based on spatial dispersion and residual weighting described above can effectively ensure the uniformity of feature point distribution.

[0120] Optionally, after step 303, the method may further include: performing the following operations for each second feature point: the electronic device calculates a gradient direction histogram within a preset pixel neighborhood (e.g., a 16×16 pixel neighborhood) centered on the second feature point, and determines the main direction, which is the mode of gradient directions within the preset pixel neighborhood (i.e., the direction with the highest frequency of occurrence). In addition, when the secondary peak exceeds 80% of the main peak, an auxiliary direction is generated, i.e., an additional descriptor is generated to enhance rotation robustness; the electronic device divides the preset pixel neighborhood into neighborhood sub-blocks (e.g., 4×4 neighborhood sub-blocks), and calculates 8 directional gradient histograms for each neighborhood sub-block, concatenating them into a 128-dimensional feature vector. After L2 normalization, a descriptor robust to rotation, scaling, and illumination is obtained for the second feature point. The goal of normalization is to eliminate the effects of illumination changes (e.g., linear changes in brightness do not alter the descriptor) and improve robustness to local contrast. Based on this, the electronic device can ultimately determine the descriptor for each second feature point. In this way, even with changes in image brightness, slight rotation, or scale, the descriptors for the same feature point will be highly similar, providing a precise basis for subsequent image registration.

[0121] The 128-dimensional dimension is calculated from 16 neighboring sub-blocks × 8 directions of each neighboring sub-block.

[0122] Optionally, the formula for calculating the above descriptor is: .

[0123] Descriptor representing the second feature point (normalized to unit length); Representing the eigenvector The Gradient magnitude in each dimension ( (∈[1,128]), calculated from the horizontal and vertical gradients of the image; Representing the eigenvector The L2 norm.

[0124] Step 304: Solve the homogeneous linear equations of the pixel pairs that match the second feature point using the Singular Value Decomposition (SVD) method to obtain the transformation homography matrix; and register the first target image to be registered according to the transformation homography matrix to obtain the target image.

[0125] Among them, the Singular Value Decomposition (SVD) method is a mathematical method that decomposes any real or complex matrix into the product of three specific matrices.

[0126] In step 304, the electronic device solves the homography matrix by decomposing it using the SVD method, which effectively eliminates the influence of noise and outliers in feature point matching, improves the geometric accuracy of image registration, and then achieves efficient alignment between the first target image to be registered and the target reference image based on the global projection transformation of the homography matrix, ensuring stable registration results under illumination and viewing angle differences, and obtaining a target image of better quality.

[0127] The following section elaborates on the use of Singular Value Decomposition (SVD) to solve the homogeneous linear equations of pixel pairs matching the second feature point in electronic devices, yielding the transformation homography matrix:

[0128] In some embodiments, the electronic device uses the Singular Value Decomposition (SVD) method to solve the homogeneous linear equations of the pixel pairs matching the second feature points to obtain the transformation homography matrix. This process may include: the electronic device determining the pixels matching each second feature point from the target reference image based on the descriptors of all second feature points and constructing pixel pairs; the electronic device constructing homogeneous equations based on the pixel coordinates of each pixel pair; and the electronic device using the SVD method to solve the homogeneous linear equations constructed from all homogeneous equations to obtain the transformation homography matrix.

[0129] In determining the transformation homography matrix, for each of the S second feature points (where S is an integer greater than 1), the electronic device determines the feature point in the target reference image that is closest to the descriptor of the second feature point as the pixel point matching that second feature point, and constructs a pixel pair between the second feature point and the pixel point, with the second feature point being the closest to the descriptor of the second feature point. Pixel coordinates of the second feature point and the pixel coordinates of the matching pixels For example, the equation can be expanded as follows:

[0130] And rewrite the equation in matrix form: ,in, , which represents a flattened vector constructed from all elements in the transformation homography matrix.

[0131] For S second feature points, the electronic device can construct S pixel pairs, and then construct S homogeneous equations to obtain a homogeneous linear equation system. Finally, the electronic device uses the SVD method to solve the homogeneous linear equation system to obtain the transformation homography matrix, which is then used to accurately register the first target image to be registered.

[0132] The following section elaborates on the use of the SVD method to solve the homogeneous linear equation system constructed from all homogeneous equations in electronic devices, and the resulting transformation homography matrix:

[0133] In some embodiments, the above homogeneous linear equation system is: , This represents the coefficient matrix.

[0134] Based on this, the electronic device uses the SVD method to solve the homogeneous linear equation system constructed from all homogeneous equations to obtain the transformation homography matrix, which may include: the electronic device using the SVD method to solve the coefficient matrix. Decomposition yields the right singular vector matrix. The electronic device will use the right singular vector matrix. The last column element in the vector is used as a flattened vector. The electronic device is based on flattened vectors. Use all elements in the matrix to construct the transformation homography matrix.

[0135] In the process of constructing the transformation homography matrix, since the electronic device has already constructed S matrix forms, it constructs a 2S×9 coefficient matrix for these S pixel pairs. The above homogeneous linear equation system is solved using the SVD method. Specifically, the coefficient matrix is... Decompose to obtain To obtain the left singular vector matrix diagonal matrix and right singular vector matrix Then, the electronic device will transfer the right singular vector matrix. The last column element in the vector is used as a flattened vector. And then based on this flattened vector Use all elements in the matrix to construct the transformation homography matrix.

[0136] It should be noted that the above coefficient matrix It is an m×n matrix; the above left singular vector matrix It is an m×m orthogonal matrix; the diagonal matrix mentioned above It is an m×n diagonal matrix; the above right singular vector matrix It is an n×n orthogonal matrix.

[0137] Optionally, after step 304, the method may further include: the electronic device iterating again using the improved RANSAC algorithm until the reprojection error is minimized to less than a preset reprojection error threshold (e.g., ...). (pixels), at this point, the optimal transformation homography matrix can be calculated using the optimal single mapping algorithm to achieve accurate registration of the first target image to be registered.

[0138] In this embodiment, the technical solutions of steps 301-304 above pre-align the image using an optimal affine transformation model, effectively reducing the computational complexity and load of the image registration process, thereby improving image registration efficiency and shortening the single-frame processing time from minutes to seconds, meeting the real-time requirements of high-frequency detection in electronic equipment. Simultaneously, the use of a Gaussian difference pyramid three-dimensional extreme value search combined with principal curvature constraints effectively filters edge noise points, and the improved RANSAC algorithm uses dual conditions (reprojection error and sampling probability) to screen feature points, effectively reducing the mismatch rate under conditions of large camera viewing angle differences and rain, fog, and low light, and improving image registration accuracy, thereby reducing the risk of missed foreign object detection. In other words, this method can break through the contradictory closed loop of "low efficiency - insufficient accuracy," enabling efficient and high-precision FOD detection, thereby improving airport security inspection efficiency and ensuring aviation operational safety.

[0139] The image registration apparatus provided in the embodiments of this application is described below. The image registration apparatus described below can be referred to in correspondence with the image registration method described above.

[0140] Figure 4 This is a schematic diagram of the image registration device provided in an embodiment of this application. Figure 4 As shown, the device includes: an acquisition module 401, a transformation module 402, a filtering module 403, and a registration module 404.

[0141] The acquisition module 401 is used to acquire the target reference image and the first target registration image corresponding to the area to be inspected containing foreign object FOD.

[0142] The transformation module 402 is used to construct an optimal affine transformation model based on the target reference image and N preset pitch angles, where N is an integer greater than 1, and to use the optimal affine transformation model to perform a linear transformation on the first target image to be registered to obtain the second target image to be registered.

[0143] The filtering module 403 is used to perform a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and to filter extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature points of the second target image to be registered; and to use an improved Random Sample Consistency (RANSAC) algorithm on all the first feature points to filter feature points whose reprojection error and sampling probability meet preset conditions as the second feature points.

[0144] The registration module 404 is used to solve the homogeneous linear equations of the pixel pairs that match the second feature point using the singular value decomposition (SVD) method to obtain the transformation homography matrix; and to register the first target image to be registered according to the transformation homography matrix to obtain the target image.

[0145] Optionally, the transformation module 402 is specifically configured to obtain the N preset pitch angles from within a preset pitch angle range; construct N affine transformation models according to the N preset pitch angles and their respective corresponding optimal azimuth angles; perform feature point detection on the target reference image using the N affine transformation models respectively, to obtain the number of feature points and the spatial distribution entropy corresponding to each affine transformation model; determine n affine transformation models from the N affine transformation models in descending order of the number of feature points, where 0 < n < N; and use the affine transformation model with the largest spatial distribution entropy among the n affine transformation models as the optimal affine transformation model.

[0146] Optionally, the Gaussian difference pyramid includes M Gaussian difference images, where M is an integer greater than 2. The M Gaussian difference images include a target Gaussian difference image, and the target Gaussian difference image is a Gaussian difference image adjacent to two Gaussian difference images. The screening module 403 is specifically configured to traverse each candidate pixel point in the target Gaussian difference image to obtain a first pixel point that is a corner point common to the candidate pixel point; determine a second pixel point corresponding to the positions of the candidate pixel point and the first pixel point from the two Gaussian difference images adjacent to the target Gaussian difference image; use the pixel point with the maximum or minimum brightness value among the candidate pixel point, the first pixel point, and the second pixel point as an extreme point; determine the principal curvature according to the Hessian matrix of the extreme point; and screen out the extreme points with principal curvatures not exceeding a preset principal curvature threshold from all the extreme points as the first feature points of the second target registration image.

[0147] Optionally, the screening module 403 is specifically configured to randomly select P1 first sub-feature points and P2 second sub-feature points from all the first feature points, where P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of the first feature points; calculate a homography matrix according to the P1 first sub-feature points; for each second sub-feature point, determine a reprojection error according to the homography matrix, the pixel coordinates of the second sub-feature point, and the pixel coordinates of the pixel point corresponding to the second sub-feature point in the second target registration image; and determine a sampling probability according to the reprojection error; screen out the second sub-feature points with reprojection errors less than a preset error threshold and sampling probabilities greater than a preset probability threshold from the P2 second sub-feature points as the second feature points.

[0148] Optionally, the registration module 404 is specifically configured to determine pixel points matching each second feature point from the target reference image according to the descriptors of all the second feature points, and construct pixel point pairs; construct homogeneous equations according to the pixel coordinates of each pixel point pair; solve the homogeneous linear equations constructed from all the homogeneous equations using the SVD method to obtain the transformation homography matrix.

[0149] Optionally, the transformation module 402 is specifically used to perform the following operations for each preset pitch angle: determine the tilt parameter based on the preset pitch angle, wherein the calculation formula for the tilt parameter is: , This indicates the tilt parameter. This represents the preset pitch angle; based on this tilt parameter, the azimuth sampling interval is determined, and the formula for calculating the azimuth sampling interval is: , This indicates the azimuth sampling interval. The constant is represented; the optimal azimuth angle is determined based on the azimuth angle sampling interval; and an affine transformation model is constructed based on the optimal azimuth angle and the preset elevation angle.

[0150] Optionally, the filtering module 403 is specifically used to filter based on the extreme point at... The first and second partial derivatives in the direction, the extreme point at The Hessian matrix is ​​constructed using the second-order partial derivative of the first-order partial derivative and the cross-partial derivative of the second-order partial derivative. Based on this Hessian matrix, the principal curvature is determined, and the formula for calculating the principal curvature is as follows: ;in, This indicates the principal curvature; This represents the Hessian matrix; This represents the summation of the first and second partial derivatives and the second partial derivative. This represents the difference between the product of the first and second partial derivatives and the square of the cross partial derivative.

[0151] Optionally, the homogeneous linear equation system is: , Represents the coefficient matrix. Represents a flattened vector constructed from all elements of the transformation homography matrix. Registration module 404 is specifically used to apply this SVD method to the coefficient matrix. Decomposition yields the right singular vector matrix. ; this right singular vector matrix The last column element in the vector is used as the flattened vector. Based on this flattened vector Use all elements in the matrix to construct the transformation homography matrix.

[0152] Optionally, the acquisition module 401 is specifically used to acquire images of the region to be inspected by the multi-camera array; use the image acquired by one camera in the multi-camera array as the first reference image and the images acquired by the remaining cameras as the first images to be registered; scale the first reference image and the first images to be registered by a preset sampling operator to obtain a second reference image and a second images to be registered; and perform high-frequency noise elimination on the second reference image and the second images to be registered by a Gaussian filter to obtain the target reference image and the first target images to be registered.

[0153] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can call logic instructions in the memory 530 to execute an image registration method, which includes: acquiring a target reference image and a first target image to be registered corresponding to the area to be inspected containing foreign object (FOD); constructing an optimal affine transformation model based on the target reference image and N preset pitch angles, where N is an integer greater than 1, and using the optimal affine transformation model to perform a linear transformation on the first target image to be registered to obtain a second target image to be registered; performing a three-dimensional extreme value search on the difference of Gaussian pyramid corresponding to the second target image to be registered, and selecting extreme points whose principal curvature does not exceed a preset principal curvature threshold as first feature points of the second target image to be registered; and using an improved random sampling consistency (RANSAC) algorithm on all first feature points to select feature points whose reprojection error and sampling probability meet preset conditions as second feature points; using the singular value decomposition (SVD) method to solve the homogeneous linear equations of pixel pairs matched with the second feature points to obtain a transformation homography matrix; and registering the first target image to be registered based on the transformation homography matrix to obtain a target image.

[0154] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image registration method provided by the above methods. The method includes: acquiring a target reference image and a first target image to be registered corresponding to the area to be inspected containing foreign object (FOD); constructing an optimal affine transformation model based on the target reference image and N preset pitch angles, where N is an integer greater than 1; and using the optimal affine transformation model to perform a linear transformation on the first target image to be registered to obtain a second target image. The target image to be registered is obtained by performing a three-dimensional extreme value search on the Gaussian difference pyramid corresponding to the second target image to be registered, and selecting extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature points of the second target image to be registered; and using an improved Random Sample Consensus (RANSAC) algorithm on all the first feature points, selecting feature points whose reprojection error and sampling probability meet preset conditions as the second feature points; using the Singular Value Decomposition (SVD) method to solve the homogeneous linear equations of the pixel pairs matched with the second feature points to obtain the transformation homography matrix; and registering the first target image to be registered according to the transformation homography matrix to obtain the target image.

[0156] In another aspect, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the image registration method provided by the methods described above. This method includes: acquiring a target reference image and a first target image to be registered, corresponding to an area to be inspected containing foreign object (FOD); constructing an optimal affine transformation model based on the target reference image and N preset pitch angles, where N is an integer greater than 1; and using the optimal affine transformation model to perform a linear transformation on the first target image to be registered, obtaining a second target image to be registered; and performing a linear transformation on the second target image... The Gaussian difference pyramid corresponding to the image to be registered performs a three-dimensional extreme value search, and selects extreme points whose principal curvature does not exceed a preset principal curvature threshold as the first feature points of the second target image to be registered; and applies an improved Random Sample Consensus (RANSAC) algorithm to all first feature points, and selects feature points whose reprojection error and sampling probability meet preset conditions as second feature points; the singular value decomposition (SVD) method is used to solve the homogeneous linear equation system of pixel pairs matched with the second feature points to obtain the transformation homography matrix; and the first target image to be registered is registered according to the transformation homography matrix to obtain the target image.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image registration method characterized by, The method comprises the steps of: acquiring a target reference image and a first target image to be registered corresponding to a region to be detected containing foreign object debris (FOD); constructing an optimal affine transformation model according to the target reference image and N preset pitch angles, N being an integer greater than 1, and performing linear transformation on the first target image to be registered by using the optimal affine transformation model to obtain a second target image to be registered; performing three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image to be registered, and screening extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered; and using an improved random sample consensus (RANSAC) algorithm on all first feature points to screen feature points with re-projection errors and sampling probabilities satisfying preset conditions as second feature points; solving a homogeneous linear equation group of pixel point pairs matched with the second feature points by using a singular value decomposition (SVD) method to obtain a transformation homography matrix, and performing registration on the first target image to be registered according to the transformation homography matrix to obtain a target image.

2. The image registration method of claim 1, wherein, The method comprises the steps of: acquiring the N preset pitch angles from a preset pitch angle range; constructing N affine transformation models according to the N preset pitch angles and respective optimal azimuth angles; performing feature point detection on the target reference image by using the N affine transformation models respectively to obtain the number and spatial distribution entropy of feature points corresponding to each affine transformation model; determining n affine transformation models from the N affine transformation models in descending order of the number of feature points, 0 < n < N; and taking an affine transformation model with the maximum spatial distribution entropy in the n affine transformation models as the optimal affine transformation model.

3. The image registration method of claim 1, wherein, The Gaussian difference pyramid comprises M Gaussian difference images, M being an integer greater than 2, and the M Gaussian difference images comprise a target Gaussian difference image, the target Gaussian difference image being a Gaussian difference image adjacent to two Gaussian difference images. The method comprises the steps of: traversing each candidate pixel point in the target Gaussian difference image to acquire a first pixel point sharing a corner point with the candidate pixel point; determining a second pixel point corresponding to the positions of the candidate pixel point and the first pixel point from two Gaussian difference images adjacent to the target Gaussian difference image; taking a pixel point with the maximum or minimum luminance value among the candidate pixel point, the first pixel point and the second pixel point as an extreme value point; and determining the principal curvature according to the Hessian matrix of the extreme value point; screening extreme value points with principal curvatures not exceeding a preset principal curvature threshold as first feature points of the second target image to be registered from all extreme value points.

4. The image registration method of claim 1, wherein, The method comprises the steps of: using an improved random sample consensus (RANSAC) algorithm on all first feature points to screen feature points with re-projection errors and sampling probabilities satisfying preset conditions as second feature points. Randomly select P1 first sub-feature points and P2 second sub-feature points from the all first feature points, P1 and P2 are both integers greater than 1, and the sum of P1 and P2 is the total number of the first feature points; According to the P1 first sub-feature points, a homography matrix is calculated; For each second sub-feature point, a re-projection error is determined according to the homography matrix, the pixel coordinates of the second sub-feature point, and the pixel coordinates of the pixel point corresponding to the second sub-feature point in the second target image to be registered; and a sampling probability is determined according to the re-projection error; From the P2 second sub-feature points, second sub-feature points with a re-projection error less than a preset error threshold and a sampling probability greater than a preset probability threshold are selected as the second feature points.

5. The image registration method of claim 1, wherein, The singular value decomposition (SVD) method is used to solve the homogeneous linear equations of the pixel point pairs matched with the second feature points to obtain a transformation homography matrix, including: According to the descriptors of all second feature points, pixel points matched with each second feature point are determined from the target reference image, and pixel point pairs are constructed; According to the pixel coordinates of each pixel point pair, a homogeneous equation is constructed; The SVD method is used to solve the homogeneous linear equations constructed by all homogeneous equations to obtain the transformation homography matrix.

6. The image registration method of claim 2, wherein, The N affine transformation models are constructed according to the N preset pitch angles and the optimal azimuth angles corresponding to each preset pitch angle, including: For each preset pitch angle, the following operations are performed: According to the preset pitch angle, a tilt parameter is determined, and a calculation formula of the tilt parameter is: , represents the tilt parameter, represents the preset pitch angle; According to the tilt parameter, a azimuth angle sampling interval is determined, and a calculation formula of the azimuth angle sampling interval is: , denotes the azimuth angle sampling interval, denotes a constant; An optimal azimuth angle is determined according to the azimuth angle sampling interval; An affine transformation model is constructed according to the optimal azimuth angle and the preset pitch angle.

7. The image registration method of claim 3, wherein, The principal curvatures are determined according to the Hessian matrix of the extreme point, including: According to the first second-order partial derivative of the extreme point in the first direction, the second second-order partial derivative of the extreme point in the second direction, and the cross partial derivative of the first second-order partial derivative and the second second-order partial derivative, the Hessian matrix is constructed; According to the Hessian matrix, the principal curvatures are determined, and a calculation formula of the principal curvatures is: ; wherein, denotes the principal curvatures; denotes the Hessian matrix; denotes the sum of the first and second second-order partial derivatives; denotes the difference between the product of the first and second second-order partial derivatives and the square of the cross partial derivative.

8. The image registration method of claim 5, wherein, The homogeneous linear equation system is , denotes the coefficient matrix, denotes a flattened vector built from all elements of the variable transform homography matrix, The SVD method is used to solve the homogeneous linear equations constructed by all homogeneous equations to obtain the transformation homography matrix, including: The SVD method is used to decompose the coefficient matrix to obtain a right singular vector matrix ; the last column element in the right singular vector matrix is taken as the flattened vector ; According to all elements in the flattening vector the warping homography is constructed.

9. The image registration method of any of claims 1-8, wherein, The target reference image and the first target image to be registered corresponding to the foreign object debris (FOD) containing area are obtained, including: Images collected by a multi-view camera array on the area to be inspected are obtained; An image collected by one camera in the multi-view camera array is taken as a first reference image, and images collected by the remaining cameras are taken as first images to be registered; The first reference image and the first image to be registered are respectively scaled by a preset sampling operator to obtain a second reference image and a second image to be registered; The second reference image and the second image to be registered are respectively subjected to high-frequency noise elimination by a Gaussian filter to obtain the target reference image and the first target image to be registered.

10. An image registration apparatus characterized by comprising: It includes: An acquisition module is configured to acquire a target reference image and a first target image to be registered corresponding to a foreign object debris (FOD) containing area to be inspected; A transformation module is configured to construct an optimal affine transformation model according to the target reference image and N preset pitch angles, N is an integer greater than 1, and the optimal affine transformation model is used to perform linear transformation on the first target image to be registered to obtain a second target image to be registered. The screening module is configured to perform a three-dimensional extreme value search on a Gaussian difference pyramid corresponding to the second target image pair to be registered, and screen extreme value points with a principal curvature not exceeding a preset principal curvature threshold as first feature points of the second target image pair to be registered; and adopt an improved random sample consensus (RANSAC) algorithm to screen feature points with a reprojection error and a sampling probability satisfying a preset condition as second feature points. The registration module is configured to adopt a singular value decomposition (SVD) method to solve a homogeneous linear equation set of pixel point pairs matched with the second feature points, and obtain a variable single homographic matrix; and perform registration on the first target image pair to be registered according to the variable single homographic matrix to obtain a target image.

Images