A method and system for registering shear wave domain heterogeneous images based on FPGA

Abstract

The present application relates to the technical field of image processing, in particular to a method and system for registering heterogeneous images in shear wave domain based on FPGA. The method for registering heterogeneous images in shear wave domain based on FPGA comprises the following steps: collecting video stream images in real time, synchronizing frame frequency and size of the video stream images across clock domains, obtaining images synchronized across clock domains, decomposing the images synchronized across clock domains through shear wave domain, obtaining low-frequency band images and high-frequency band images, preliminarily screening the low-frequency band images and the high-frequency band images, obtaining screened images, registering the screened images through fast Fourier Mellin transform, obtaining registered images, and outputting video stream in real time by matching and superimposing the registered images. The present application improves the robustness of the algorithm through the scale invariance and multi-scale characteristics of shear wave domain feature screening; the fast Fourier Mellin transform effectively overcomes the oversampling problem in the low-frequency region, and improves the accuracy and efficiency of registration.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for cross-source image registration in the sheared wave domain based on FPGA. Background Technology

[0002] Modern intelligent weaponry, such as unmanned reconnaissance aircraft, cruise missiles, and satellite detection devices, can no longer rely on a single data source to provide complete data. Multiple sensors are used to collect data on the same scene or target, combining their characteristics to create a new image. The complementary information from different sensors provides a multifaceted description of the target's features, facilitating observation and subsequent processing. However, heterogeneous images often exhibit spatial variations, different imaging principles, and different image resolutions. The purpose of heterogeneous image registration is to extract the correlation between heterogeneous images based on their differences, and to find the optimal transformation matrix between them.

[0003] Existing traditional image registration techniques struggle to design universal methods suitable for visual perception tasks due to the diversity of images to be registered and various types of degradation. They must consider not only assumed geometric deformation types between images but also radiation distortion and noise damage, requiring high registration accuracy and adapting to the characteristics of the applied data. Newer deep learning-based image registration methods regress and predict image transformation relationships by learning multi-scale, high-dimensional semantic features. Compared to traditional methods, these methods achieve more significant results because deep neural networks can learn more effective features from large amounts of labeled data. However, deep learning relies on data annotation and training in images, requiring substantial human resources, which is a major obstacle to the widespread application of deep neural network-based registration methods. Furthermore, the processing flow is complex and practical deployment is difficult. Currently used methods based on traditional feature registration or deep learning registration are mostly PC post-processing-based, making it difficult to achieve high-precision, robust real-time heterogeneous video stream registration at edge devices such as unmanned reconnaissance aircraft, cruise missiles, and satellite detection equipment. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a sheared wave domain heterogeneous image registration method and system based on FPGA, so as to achieve high-precision and robust heterogeneous image registration of targets such as ships at the edge of the target at high registration speed.

[0005] To achieve the above and other related objectives, this invention provides an FPGA-based method for heterogeneous image registration in the sheared wave domain, comprising:

[0006] S1. Real-time acquisition of video stream images, synchronization of the frame rate and size of the video stream images across clock domains, to obtain images synchronized across clock domains;

[0007] S2. The image after cross-clock domain synchronization is decomposed by shearing the wave domain to obtain a low-frequency band image and a high-frequency band image, and the low-frequency band image and the high-frequency band image are initially screened to obtain a screened image;

[0008] S3. The filtered images are registered using Fast Fourier Merlin Transform to obtain the registered images. The registered images are then matched and superimposed to output a video stream in real time.

[0009] In one embodiment of the present invention, the video stream image is a dual-channel video stream image consisting of visible light and infrared light.

[0010] In one embodiment of the invention, the frame rate and size of the video stream image are synchronized with the bus clock by using an asynchronous first-in-first-out (FIFO) queue.

[0011] In one embodiment of the present invention, the image synchronized across the clock domain is decomposed by shearing the wave domain to obtain a low-frequency band image and a high-frequency band image, and the low-frequency band image and the high-frequency band image are initially screened to obtain a screened image including:

[0012] S21. The visible light and infrared dual-channel video stream images are decomposed into multi-scale images through non-downsampling shear wave transform to obtain the low-frequency band image and the high-frequency band image.

[0013] S22. At each decomposition level of the low-frequency band image and the high-frequency band image, non-subsampled pyramid group (NSPF) is used to obtain low-frequency components and high-frequency components. The low-frequency components and high-frequency components are decomposed again to obtain the low-frequency components of the image singularities for iterative decomposition, resulting in j+1 sub-images and j high-frequency images of the same size as the original image, i.e., the video stream image, where j is the decomposition level. Then, l-level multi-directional decomposition is performed.

[0014] S23. Each high-frequency component is decomposed into 2l sub-band images of the same size by using a shearing filter bank;

[0015] S24. Perform SURF feature point calculations on the original image, the decomposed low-frequency band image, and the high-frequency band image respectively to obtain the calculated feature points for preliminary image screening.

[0016] In one embodiment of the present invention, the step of performing SURF feature point calculations on the original image, the decomposed low-frequency band image, and the high-frequency band image respectively to obtain the calculated feature points for preliminary image screening includes:

[0017] The Hessian matrix is ​​constructed using the SURF algorithm to obtain the feature points of the image;

[0018] The image is blurred using a box filter, and the error of the feature points is balanced by weighting coefficients.

[0019] The Ransac feature matching algorithm is used to describe the Euclidean distance between the sub-band images using balanced feature points, in order to perform preliminary image screening.

[0020] In one embodiment of the present invention, the step of registering the filtered images using Fast Fourier Merlin Transform to obtain a registered image, and then outputting a video stream in real time by matching and superimposing the registered images, includes:

[0021] The Fourier spectrum of the filtered image is transformed into polar coordinates using an adaptive resampling method, and the number of sampling points is controlled by the polar radius length.

[0022] The visible light dual-channel video stream images are converted from Cartesian coordinates to adaptive polar coordinates, starting from the center of a circle with a radius of R. max The circular surface with a radius of -1 is used as the effective region for coordinate transformation. The remaining portion of the visible light dual-channel video stream image, excluding the effective region, is discarded. Where R... max It is 1 / 4 of the shortest side of the visible light dual-channel video stream image;

[0023] Starting from a point on the negative half of the polar radius r-axis at the center of the circle, uniformly sample the circumference corresponding to the radius in a clockwise direction as the radius increases, until the entire radius is R. i Circular sampling is complete;

[0024] The rotation factor θ, scale factor a, and translation (x0, y0) of the relative infrared reference map of the visible light dual-channel video stream image are calculated using Fast Fourier Merlin Transform.

[0025] The image f1(x,y) and the image f2(x,y) are registered according to f2(x,y)=f1(a(xcosθ+ysinθ)+x0(a(-xsinθ+ycosθ))+y0), and the registered visible light image is obtained.

[0026] The registered visible light image is superimposed with the infrared dual-channel video stream image to output the video stream in real time.

[0027] In one embodiment of the present invention, the visible light and infrared dual-channel video stream images are transmitted in VGA timing format with a frame rate of 50 FPS.

[0028] This invention also provides an FPGA-based sheared wave domain heterogeneous image registration system, comprising:

[0029] Cameras are used to capture video stream images in real time;

[0030] An FPGA is used to synchronize the frame rate and size of the video stream image across clock domains to obtain an image synchronized across clock domains.

[0031] The FPGA is also used to decompose the image after cross-clock domain synchronization by shearing the wave domain to obtain low-frequency band image and high-frequency band image, and to perform preliminary screening on the low-frequency band image and high-frequency band image to obtain the screened image;

[0032] The FPGA is also used to register the filtered images using Fast Fourier Merlin Transform to obtain a registered image. By matching and superimposing the registered images, a video stream is output in real time.

[0033] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the above-described FPGA-based sheared wave domain heterogeneous image registration method.

[0034] As described above, the FPGA-based sheared wave domain heterogeneous image registration method and system of the present invention have the following beneficial effects:

[0035] (1) The FPGA-based shearing domain heterogeneous image registration method of the present invention acquires video stream images in real time, synchronizes the frame rate and size of the video stream images across clock domains to obtain synchronized images across clock domains; decomposes the synchronized images across clock domains to obtain low-frequency band images and high-frequency band images, and performs preliminary screening on the low-frequency band images and high-frequency band images to obtain screened images; registers the screened images using fast Fourier Merlin transform to obtain registered images; and outputs the video stream in real time by matching and superimposing the registered images.

[0036] (2) This invention improves the robustness of the algorithm by using scale invariance and multi-scale characteristics of shear wave domain feature screening; it effectively overcomes the oversampling problem in the low frequency region by using fast Fourier Merlin transform, thereby improving the accuracy and efficiency of registration; it accelerates the entire process of the algorithm on FPGA, and can achieve high-precision and robust heterogeneous image registration at the edge with high registration speed, and output heterogeneous image registration in real time.

[0037] (3) The FPGA-based shearing wave domain heterogeneous image registration method of the present invention is not only applicable to ship image registration, but also to image registration in other fields, such as medical image registration, remote sensing image registration, etc., and has wide application value and scalability. Attached Figure Description

[0038] Figure 1 A flowchart illustrating the workflow of the FPGA-based sheared wave domain heterogeneous image registration method provided in this application embodiment.

[0039] Figure 2 This is a schematic diagram of the shearing wave domain transformation process of the FPGA-based heterogeneous image registration method provided in an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of the structure of an FPGA-based sheared wave domain heterogeneous image registration system provided in an embodiment of the present invention. Detailed Implementation

[0041] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0042] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0043] Please see Figure 1 , Figure 1 This is a flowchart illustrating the workflow of the FPGA-based sheared wave domain heterogeneous image registration method provided in this application embodiment. The present invention provides an FPGA-based sheared wave domain heterogeneous image registration method, comprising:

[0044] S1. Real-time acquisition of video stream images, synchronization of the frame rate and size of the video stream images across clock domains, to obtain images synchronized across clock domains;

[0045] S2. The image after cross-clock domain synchronization is decomposed by shearing the wave domain to obtain a low-frequency band image and a high-frequency band image, and the low-frequency band image and the high-frequency band image are initially screened to obtain a screened image;

[0046] S3. The filtered images are registered using Fast Fourier Merlin Transform to obtain the registered images. The registered images are then matched and superimposed to output a video stream in real time.

[0047] The FPGA-based heterogeneous image registration method of this invention can improve the robustness of the algorithm by using the scale invariance and multi-scale characteristics of shear domain feature screening; it effectively overcomes the oversampling problem in the low-frequency region by using Fast Fourier Merlin Transform, thereby improving the accuracy and efficiency of registration; and it accelerates the entire algorithm process on FPGA, enabling high-precision and robust heterogeneous image registration at the edge while maintaining high registration speed, and providing real-time heterogeneous image registration output.

[0048] Specifically, in step S1, the video stream image is a dual-channel video stream image consisting of visible light and infrared. The frame rate and size of the dual-channel visible light and infrared video stream images are synchronized with the bus clock using an asynchronous first-in-first-out (FIFO) queue. The dual-channel visible light and infrared video stream images are transmitted in VGA timing format with a frame rate of 50 FPS. Line valid and frame valid signals are used to indicate the data position.

[0049] In this embodiment, the cross-clock domain synchronization frame frequency of the image after cross-clock domain synchronization introduces an external 100MHz clock into the clock management MMCM and multiplies it to 200MHz, dividing it into 3 related clock domains to drive the underlying data processing modules. The AXI4 bus and the top-level state machine are driven by a 100MHz clock provided by the on-chip ARM system. It uses synchronous access to the register table, while large data transmission uses asynchronous access. By controlling the READY signal for synchronization, the frame frequency synchronization of the visible light and infrared asynchronous video stream processing is ensured.

[0050] In this embodiment, the synchronization size can be, but is not limited to, using bilinear interpolation to scale the visible light dual-channel video stream image to the same size as the infrared dual-channel video stream image in real time.

[0051] Specifically, in step S2, the image synchronized across the clock domain is decomposed by shearing the wave domain to obtain a low-frequency band image and a high-frequency band image. The low-frequency band image and the high-frequency band image are then preliminarily filtered to obtain the filtered images, which include:

[0052] S21. The visible light and infrared dual-channel video stream images are decomposed into multi-scale images through non-downsampling shear wave transform to obtain the low-frequency band image and the high-frequency band image.

[0053] S22. At each decomposition level of the low-frequency band image and the high-frequency band image, non-subsampled pyramid group (NSPF) is used to obtain low-frequency components and high-frequency components. The low-frequency components and high-frequency components are decomposed again to obtain the low-frequency components of the image singularities for iterative decomposition, resulting in j+1 sub-images and j high-frequency images of the same size as the original image, i.e., the video stream image, where j is the decomposition level. Then, l-level multi-directional decomposition is performed.

[0054] S23. Each high-frequency component is decomposed into 2l sub-band images of the same size by using a shearing filter bank;

[0055] S24. Perform SURF feature point calculation on the original image, the decomposed low-frequency band image and the high-frequency band image respectively to obtain the calculated feature points for preliminary image screening.

[0056] The Hessian matrix is ​​constructed using the SURF algorithm to obtain the feature points of the image;

[0057] The image is blurred using a box filter, and the error of the feature points is balanced by weighting coefficients.

[0058] The Ransac feature matching algorithm is used to describe the Euclidean distance between the sub-band images using balanced feature points, in order to perform preliminary image screening.

[0059] Specifically, the low-frequency and high-frequency images are decomposed into sub-bands of different scales and orientations. Then, a local transformation is performed on each sub-band. The sub-band images include both low-frequency and high-frequency sub-band images. The low-frequency sub-band image contains most of the image's energy, while the high-frequency sub-band contains detailed and texture information of the image's targets.

[0060] Please see Figure 2 , Figure 2 This is a schematic diagram of the shearing domain transformation process of the FPGA-based heterogeneous image registration method provided in this embodiment of the invention. By decomposing each high-frequency component into 2l coefficient components of equal size using a shearing filter bank, shift invariance is satisfied by avoiding subsampling.

[0061] Specifically, the Hessian matrix is ​​a basic linear smoothing filter, also known as a mean filter.

[0062] Specifically, the Euclidean distance between the sub-band images is described using balanced feature points to perform preliminary image screening. The screened images are robust to noise and other interference and can adapt to changes in ship images in different scenes and environments, including changes in illumination, angle, and scale.

[0063] Specifically, in step S3, the process of registering the filtered images using Fast Fourier Merlin Transform to obtain a registered image, and then outputting the video stream in real time by matching and superimposing the registered images, includes:

[0064] The Fourier spectrum of the filtered image is transformed into polar coordinates using an adaptive resampling method, and the number of sampling points is controlled by the polar radius length.

[0065] The visible light dual-channel video stream images are converted from Cartesian coordinates to adaptive polar coordinates, starting from the center of a circle with a radius of R. max The circular surface with a radius of -1 is used as the effective region for coordinate transformation. The remaining portion of the visible light dual-channel video stream image, excluding the effective region, is discarded. Where R... max It is 1 / 4 of the shortest side of the visible light dual-channel video stream image;

[0066] Starting from a point on the negative half of the polar radius r-axis at the center of the circle, uniformly sample the circumference corresponding to the radius in a clockwise direction as the radius increases, until the entire radius is R. i Circular sampling is complete;

[0067] The rotation factor θ, scale factor a, and translation (x0, y0) of the relative infrared reference map of the visible light dual-channel video stream image are calculated using Fast Fourier Merlin Transform.

[0068] The image f1(x,y) and the image f2(x,y) are registered according to f2(x,y)=f1(a(xcosθ+ysinθ)+x0(a(-xsinθ+ycosθ))+y0), and the registered visible light image is obtained.

[0069] The registered visible light image is superimposed with the infrared dual-channel video stream image to output the video stream in real time.

[0070] In this embodiment, an adaptive resampling method is introduced to perform polar coordinate transformation on the Fourier spectrum, and the number of sampling points is controlled by the polar radius length, thereby effectively overcoming the oversampling problem in the low-frequency region.

[0071] In this embodiment, uniform sampling is used on the polar radius, and the number of sampling points in the polar angle direction is dynamically adjusted. Since the low-frequency region, i.e. the central region of the image, is not sensitive to changes in the rotation angle, the small number of sampling points in this region has little impact on the accuracy of the rotation angle calculation. This can reduce the amount of calculation and improve the calculation speed while ensuring accuracy.

[0072] In this embodiment, by combining non-subsampled shear wave transform and fast Fourier Merlin transform, both the detailed information and geometric transformation information of the image are taken into account. By extracting the rich texture features and detailed information of the ship target image and combining them with geometric transformation matching, more accurate and robust image registration can be achieved.

[0073] Specifically, the non-subsampled shear wave transform and fast Fourier Merlin transform are implemented in hardware on the FPGA, and the real-time performance and efficiency of the algorithm are achieved through parallel computing and pipeline technology.

[0074] Specifically, the non-subsampled shear wave transform consists of a filtering module and a shear domain transform module; the fast Fourier-Melin transform consists of a Fourier transform module, a polar coordinate transformation module, and a translation / rotation module. Data is transferred between modules via an AXI bus interface, enabling pipelined data operation. The real-time processing speed and high parallel computing capabilities on the FPGA enable real-time registration and processing of ship images, meeting the needs of applications such as marine monitoring, navigation, and target recognition.

[0075] This invention also provides a specific implementation of an FPGA-based shearing domain heterogeneous image registration system. Since the FPGA-based shearing domain heterogeneous image registration system provided by this invention corresponds to the aforementioned specific implementation of the FPGA-based shearing domain heterogeneous image registration method, the FPGA-based shearing domain heterogeneous image registration system can achieve the purpose of this invention by executing the process steps in the above-mentioned method implementation. Therefore, the explanations and descriptions in the above-mentioned FPGA-based shearing domain heterogeneous image registration method implementation also apply to the specific implementation of the FPGA-based shearing domain heterogeneous image registration system provided by this invention, and will not be repeated in the following specific implementations of this invention.

[0076] Specifically, the FPGA-based sheared wave domain heterogeneous image registration system includes:

[0077] Cameras are used to capture video stream images in real time;

[0078] An FPGA is used to synchronize the frame rate and size of the video stream image across clock domains to obtain an image synchronized across clock domains.

[0079] The FPGA is also used to decompose the image after cross-clock domain synchronization by shearing the wave domain to obtain low-frequency band image and high-frequency band image, and to perform preliminary screening on the low-frequency band image and high-frequency band image to obtain the screened image;

[0080] The FPGA is also used to register the filtered images using Fast Fourier Merlin Transform to obtain a registered image. The registered image is then matched and superimposed to output a video stream in real time.

[0081] Specifically, the camera can be a dual-light camera that can acquire visible light and infrared dual-channel video stream images in real time.

[0082] Specifically, the FPGA may include, but is not limited to, a video acquisition synchronization module, a fast registration module, and a fusion output module. The video acquisition synchronization module is used to acquire visible light and infrared dual-channel video stream images and synchronize the frame rate and size of the video stream images. The fast registration module includes shearing wave domain feature filtering and fast Fourier Merlin transform to obtain the rotation, scale factor, and translation of the visible light dual-channel video stream images relative to the infrared reference image. The fusion output module is used to superimpose the visible light and infrared dual-channel video stream images after rotation, scaling, and translation transformations and output them in real time in the form of a video stream.

[0083] Specifically, the video acquisition synchronization module, fast registration module, and fusion output module are implemented based on FPGA, forming different functional IPs. All data channels adopt a fully pipelined design and are equipped with simple and independent data channel controllers. The bottom-level data channels are divided into three independent clock domains, and a dual-port FIFO is used to ensure video data synchronization. Cross-clock domains address issues such as metastability, data convergence, multi-channel fan-out, data loss, and asynchronous reset. In terms of vertical control, the bottom-level data channel controller adopts a "flat" management mechanism, and the top-level modules implement the algorithm flow by combining the bottom-level pipeline.

[0084] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the above-described FPGA-based sheared wave domain heterogeneous image registration method.

[0085] This invention provides an FPGA-based method for heterogeneous image registration in the sheared wave domain, which acquires video stream images in real time, synchronizes the frame rate and size of the video stream images across clock domains to obtain synchronized images. The synchronized images are then decomposed in the sheared wave domain to obtain low-frequency and high-frequency images, which are preliminarily filtered to obtain filtered images. Fast Fourier-Melin Transform (FFT) is used to register the filtered images to obtain registered images. The registered images are then matched and superimposed to output the video stream in real time. Therefore, this invention improves the robustness of the algorithm by leveraging the scale invariance and multi-scale characteristics of sheared wave domain feature filtering; it effectively overcomes the oversampling problem in the low-frequency region using FFT, improving registration accuracy and efficiency; and it accelerates the entire algorithm process on an FPGA, enabling high-precision, robust heterogeneous image registration at the edge while maintaining high registration speed, with real-time heterogeneous image registration output.

[0086] Meanwhile, the FPGA-based shearing domain heterogeneous image registration method of the present invention is not only applicable to ship image registration, but also to image registration in other fields, such as medical image registration and remote sensing image registration, and has broad application value and scalability.

[0087] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for registering heterogeneous images in the sheared wave domain based on FPGA, characterized in that, include: S1. Real-time acquisition of video stream images, synchronization of the frame rate and size of the video stream images across clock domains, to obtain images synchronized across clock domains; S2. The image after cross-clock domain synchronization is decomposed by shearing the wave domain to obtain a low-frequency band image and a high-frequency band image, and the low-frequency band image and the high-frequency band image are initially screened to obtain a screened image; S3. The filtered images are registered using Fast Fourier Merlin Transform to obtain a registered image. The registered images are then matched and superimposed to output a video stream in real time. The frame rate and size of the video stream image are synchronized with the bus clock by using an asynchronous first-in-first-out (FIFO) queue. The video stream image is a dual-channel video stream image consisting of visible light and infrared light; The image synchronized across the clock domain is decomposed by shearing the wave domain to obtain a low-frequency band image and a high-frequency band image. The low-frequency band image and the high-frequency band image are then preliminarily filtered to obtain the filtered images, which include: S21. The visible light and infrared dual-channel video stream images are decomposed into multi-scale images through non-downsampling shear wave transform to obtain the low-frequency band image and the high-frequency band image. S22. At each decomposition level of the low-frequency band image and the high-frequency band image, non-subsampled pyramid grouping (NSPF) is used to obtain low-frequency components and high-frequency components. The low-frequency components and high-frequency components are then decomposed again to obtain the low-frequency components of image singularities for iterative decomposition, resulting in an image of the same size as the original image, i.e., the video stream image. Sub-images and A high-frequency image, To decompose into layers, then perform... Multi-directional decomposition; S23. Decompose each of the high-frequency components into [the following] using a shearing filter bank. Sub-band images of the same size; S24. Perform SURF feature point calculation on the original image, the decomposed low-frequency band image and the high-frequency band image respectively to obtain the calculated feature points for preliminary image screening. The step of performing SURF feature point calculations on the original image, the decomposed low-frequency band image, and the high-frequency band image respectively to obtain the calculated feature points for preliminary image screening includes: The Hessian matrix is ​​constructed using the SURF algorithm to obtain the feature points of the image; The image is blurred using a box filter, and the error of the feature points is balanced by weighting coefficients. The Ransac feature matching algorithm is used to describe the Euclidean distance between the sub-band images using balanced feature points, in order to perform preliminary image screening.

2. The FPGA-based method for heterogeneous image registration in the sheared wave domain according to claim 1, characterized in that, The process of registering the filtered images using Fast Fourier Merlin Transform to obtain a registered image, and then outputting a video stream in real time by matching and superimposing the registered images, includes: The Fourier spectrum of the filtered image is transformed into polar coordinates using an adaptive resampling method, and the number of sampling points is controlled by the polar radius length. The visible light dual-channel video stream images are converted from Cartesian coordinates to adaptive polar coordinates, starting from the center of the circle with a radius of... The circular surface is used as the effective region for coordinate transformation. The remaining portion of the visible light dual-channel video stream image, excluding the effective region, is discarded. It is 1 / 4 of the shortest side of the visible light dual-channel video stream image; Starting from a point on the negative half of the polar radius r-axis from the center, the circumference corresponding to the radius is uniformly sampled clockwise as the radius increases, until the entire radius is... Circular sampling is complete; The rotation factor of the relative infrared reference image of the visible light dual-channel video stream image is calculated using Fast Fourier Merlin Transform. Scale factor Horizontal displacement ; according to To achieve image and images Registration between the two elements yields a registered visible light image; The registered visible light image is superimposed with the infrared dual-channel video stream image to output the video stream in real time.

3. The FPGA-based method for heterogeneous image registration in the sheared wave domain according to claim 1, characterized in that: The visible light and infrared dual-channel video stream images are transmitted in VGA timing format with a frame rate of 50 FPS.

4. A sheared wave domain heterogeneous image registration system based on FPGA, characterized in that, include: Cameras are used to capture video stream images in real time; An FPGA is used to synchronize the frame rate and size of the video stream image across clock domains to obtain an image synchronized across clock domains. The FPGA is also used to decompose the image after cross-clock domain synchronization by shearing the wave domain to obtain low-frequency band image and high-frequency band image, and to perform preliminary screening on the low-frequency band image and high-frequency band image to obtain the screened image; The FPGA is also used to register the filtered images using Fast Fourier Merlin Transform to obtain a registered image. By matching and superimposing the registered images, a video stream is output in real time. It is also used to synchronize the frame rate and size of the video stream image with the bus clock by using an asynchronous first-in-first-out (FIFO) queue; The video stream image is a dual-channel video stream image consisting of visible light and infrared light; The image synchronized across the clock domain is decomposed by shearing the wave domain to obtain a low-frequency band image and a high-frequency band image. The low-frequency band image and the high-frequency band image are then preliminarily filtered to obtain the filtered images, which include: The visible light and infrared dual-channel video stream images are decomposed into multi-scale images through non-downsampled shear wave transform to obtain the low-frequency band image and the high-frequency band image. At each decomposition level of the low-frequency and high-frequency images, non-subsampled pyramid grouping (NSPF) is used to obtain low-frequency and high-frequency components. These components are then decomposed again, and the low-frequency components of image singularities are used for iterative decomposition to obtain an image of the same size as the original image, i.e., the video stream image. Sub-images and A high-frequency image, To decompose into layers, then perform... Multi-directional decomposition; Each high-frequency component is decomposed into its constituent parts using a shearing filter bank. Sub-band images of the same size; The original image, the decomposed low-frequency band image, and the high-frequency band image are respectively subjected to SURF feature point calculation to obtain the calculated feature points for preliminary image screening. The step of performing SURF feature point calculations on the original image, the decomposed low-frequency band image, and the high-frequency band image respectively to obtain the calculated feature points for preliminary image screening includes: The Hessian matrix is ​​constructed using the SURF algorithm to obtain the feature points of the image; The image is blurred using a box filter, and the error of the feature points is balanced by weighting coefficients. The Ransac feature matching algorithm is used to describe the Euclidean distance between the sub-band images using balanced feature points, in order to perform preliminary image screening.

5. An electronic device comprising a processor and a memory, the memory storing program instructions, characterized in that: The processor executes program instructions to implement the FPGA-based sheared wave domain heterogeneous image registration method as described in any one of claims 1 to 3.

Images