Real-time multi-focus image fusion method and system based on FPGA

Abstract

The invention provides a real-time multi-focus image fusion method and system based on an FPGA, and relates to the technical field of image processing, and the method comprises the steps: collecting binocular images with different focal lengths, and carrying out the preprocessing of the binocular images; extracting feature points of the binocular image through a scale invariant feature transformation algorithm; calculating gradient direction distribution information of a surrounding area of the feature point through a gradient function, and determining a main direction of the feature point; calculating a gradient modulus and a gradient direction in a neighborhood of each feature point, and rotating a neighborhood region according to a main direction of the feature point to generate a descriptor; the similarity between the feature descriptors is calculated through the Manhattan distance, feature point matching is carried out, and then the matching relation of the images is determined; and splicing the registered images to obtain a fused image. According to the method, the feature points with invariance in scale, rotation and illumination change in the binocular image are extracted through the scale invariant feature transformation algorithm in combination with the gradient function, and the perception and decision-making performance of the system in a complex dynamic environment is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a real-time multi-focus image fusion method and system based on FPGA. Background Art

[0002] In autonomous driving systems, especially pure vision solutions, vehicles are usually equipped with cameras with multiple perspectives to obtain panoramic or surround view images.

[0003] However, when traditional autonomous driving systems process images, different cameras are affected by differences in spatial distribution and participating parameters. Direct fusion can cause geometric distortion and unnatural stitching artifacts at the edges of the fused image, reducing overall image clarity and visual consistency. Analyzing these blurred, stitched images requires more time for recognition and judgment, increasing decision-making latency. Unnatural stitching artifacts can also be misinterpreted by the system as actual object edges or obstacles, leading to erroneous decisions. Summary of the Invention

[0004] The purpose of the present invention is to provide a real-time multi-focus image fusion method and system based on FPGA to solve the problem mentioned in the above background technology that traditional autonomous driving systems will cause splicing marks when performing image fusion, thereby causing erroneous decisions.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a real-time multi-focus image fusion method based on FPGA, comprising the following steps: acquiring binocular images of different focal lengths and preprocessing the binocular images; extracting feature points of the binocular images using a scale-invariant feature transformation algorithm; calculating gradient direction distribution information of the area around the feature points using a gradient function to determine the main direction of the feature points; calculating the gradient modulus and gradient direction within the neighborhood of each feature point, and rotating the neighborhood area according to the main direction of the feature point to generate a descriptor; calculating the similarity between feature descriptors using Manhattan distance to match feature points, thereby determining the matching relationship of the images; and splicing the registered images to obtain a fused image.

[0006] Optionally, the preprocessing step includes: performing a smoothing operation on the binocular image through a Gaussian filter to remove noise.

[0007] Optionally, the step of extracting feature points of the binocular image through a scale-invariant feature transformation algorithm specifically includes: filtering the binocular image through a multi-scale Gaussian kernel to generate a multi-scale image set and then constructing a scale space; differentiating images of adjacent scales in the scale space to obtain a Gaussian difference image, and selecting local extreme points in the Gaussian difference image as feature points.

[0008] Optionally, the gradient function is a Tenengrad gradient function, which uses templates in the 0° / 180° direction, 45° / 225° direction, 90° / 270° direction, and 135° / 315° direction to calculate the gradient distribution information in four directions of the area around the feature point.

[0009] Optionally, the neighborhood area is divided into a plurality of neighborhood sub-areas, and the gradient information of the neighborhood sub-areas is calculated and concatenated to generate feature descriptors.

[0010] Optionally, the step of stitching the registered images specifically includes: dividing the registered images into regions according to structural features and noise features of different regions, and selecting different fusion strategies for different regions of the image.

[0011] Optionally, the step of dividing the registered image into regions according to the structural features and noise features of different regions specifically includes: calculating the clarity characteristic value of each feature point in the registered image and comparing it with a preset threshold, dividing the image into a high structural feature significant low noise area, a medium structural feature significant noise interference area and a low structural feature significant high noise area, wherein the clarity characteristic value includes the peak signal-to-noise ratio and the standard deviation.

[0012] Optionally, the step of selecting different fusion strategies for different areas of the image specifically includes: for areas with significant low noise and high structural features, selecting the clearest image sequence as the corresponding part of the fused image; for areas with significant noise interference and medium structural features, using the local weighted averaging method to perform weighted averaging processing on the pixels in the local area of the image; for areas with significant high noise and low structural features, using the neighboring area information to replace the current area information.

[0013] Optionally, the step of stitching the registered images also includes: using gradient templates in four different directions to capture gradient information of the image, performing convolution operations on each pixel in the image with the gradient templates in four directions in turn, performing square operations and square root operations on the convolution results in each direction, screening out calculation results that meet a preset gradient threshold and accumulating them to generate an image clarity evaluation function value; using the OTSU adaptive threshold algorithm to count the frequency of each gray level in the image, and then calculating the probability and mean of the foreground and background, as well as the mean of the entire image, by calculating the inter-class variance, finding the gray level that maximizes the inter-class variance among different gray levels as the gray level threshold, and dividing the image into foreground and background according to the gray level threshold.

[0014] On the other hand, the present invention also provides a real-time multi-focus image fusion system based on FPGA, including: an acquisition module, used to acquire binocular images of different focal lengths and pre-process the binocular images; a feature point extraction module, used to extract feature points of the binocular images through a scale-invariant feature transformation algorithm; a gradient function module, used to calculate the gradient direction distribution information of the area around the feature point through a gradient function, and determine the main direction of the feature point; a descriptor generation module, used to calculate the gradient modulus and gradient direction in the neighborhood of each feature point, and rotate the neighborhood area according to the main direction of the feature point to generate a descriptor; a matching relationship determination module, used to calculate the similarity between feature descriptors through Manhattan distance to perform feature point matching, and then determine the matching relationship of the image; a fusion module, used to splice the aligned images to obtain a fused image.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] This application uses FPGA architecture to adapt to multi-image integration technology, and constructs scale space and Gaussian pyramid through the scale-invariant feature transform algorithm SIFT to make the features scale-invariant; combines the Tenengrad gradient function to calculate the gradient direction distribution information of the area around the feature point, determines the main direction of the feature point, and then generates a descriptor, so that the feature point can remain unchanged in the face of rotation changes. Even if the image angle changes, the feature point can still achieve stable matching and have rotation invariance; by calculating descriptors with gradient information that is robust to lighting changes, it can overcome lighting interference and make the features lighting invariant; in complex traffic scenarios, such as urban streets with changeable light and dense vehicles and pedestrians, this application can stably match images captured by different cameras even if the lighting and image angles change, and eliminate perspective deviation to achieve accurate alignment and fusion of images, significantly improving the system's perception and decision-making performance in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the process steps of the present invention.

[0018] Figure 2 This is a block diagram of the combination of the SIFT algorithm and the gradient function of the present invention.

[0019] Figure 3 This is a schematic diagram of the extreme point search of the present invention.

[0020] Figure 4 Schematic diagram of the method for generating descriptors according to the present invention.

[0021] Figure 5 Schematic diagram of dynamic scene adaptation fusion of multi-focus images in the present invention.

[0022] Figure 6Schematic diagram of the system structure of the present invention.

[0023] In the figure: 10-acquisition module, 20-feature point extraction module, 30-gradient function module, 40-descriptor generation module, 50-matching relationship determination module, 60-fusion module. DETAILED DESCRIPTION

[0024] The following will provide a clear and complete description of the solutions of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0027] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0028] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] Please refer to Figures 1-6 The present invention provides a real-time multi-focus image fusion method based on FPGA, comprising the following steps: acquiring binocular images of different focal lengths and preprocessing the binocular images; extracting feature points of the binocular images using a scale-invariant feature transformation algorithm; calculating gradient direction distribution information of the area surrounding the feature points using a gradient function to determine the main direction of the feature points; calculating the gradient modulus and gradient direction within the neighborhood of each feature point, and rotating the neighborhood area according to the main direction of the feature point to generate a descriptor; calculating the similarity between feature descriptors using Manhattan distance to match feature points, thereby determining the matching relationship of the images; and splicing the registered images to obtain a fused image.

[0031] Specifically, the present application builds a heterogeneous visual acquisition system, in which the main camera unit and the auxiliary unit are respectively equipped with short-focus and long-focus lenses to collect image information of different focal lengths. The FPGA architecture is adapted to the multi-image integration technology, and the scale space and Gaussian pyramid are constructed through the scale-invariant feature transform algorithm SIFT to make the features scale-invariant; the gradient direction distribution information of the area around the feature point is calculated by combining the Tenengrad gradient function, the main direction of the feature point is determined, and then a descriptor is generated, so that the feature point can remain unchanged in the face of rotation changes. Even if the image angle changes, the feature point can still be stably matched and have rotation invariance; by calculating the descriptor with gradient information that is robust to illumination changes, it can overcome illumination interference and make the features illumination invariant; in complex traffic scenarios, such as urban streets with changeable light and dense vehicles and pedestrians, even if the illumination and image angles change, the present application can stably match images collected by different cameras, eliminate perspective deviation, and achieve accurate alignment and fusion of images, significantly improving the system's perception and decision-making performance in complex dynamic environments.

[0032] In some embodiments, the preprocessing step includes: performing a smoothing operation on the binocular image using a Gaussian filter to remove noise.

[0033] Specifically, the collected binocular images first undergo preliminary preprocessing, and a Gaussian filter is used to smooth the images to remove noise and reduce interference factors in subsequent processing, laying a solid foundation for accurate feature extraction and ensuring that effective information in the images can be accurately captured.

[0034] In some embodiments, the step of extracting feature points of the binocular image through a scale-invariant feature transformation algorithm specifically includes: filtering the binocular image through a multi-scale Gaussian kernel to generate a multi-scale image set and then constructing a scale space; differentiating images of adjacent scales in the scale space to obtain a Gaussian difference image, and selecting local extreme points in the Gaussian difference image as feature points.

[0035] Specifically, the input image is filtered using a multi-scale Gaussian kernel to generate a series of images at different scales, thereby constructing a scale space. By deeply analyzing how the image changes at different scales, key points with distinct scale characteristics are successfully extracted, and these features possess remarkable stability. After generating the scale space image, images at adjacent scales are differentiated to obtain a Gaussian difference image (DOG). Within this Gaussian difference image, local extreme points are identified, which are potential feature points.

[0036] Specifically, this application introduces an image matching algorithm based on the scale-invariant feature transform (SIFT), and combines it with the Tenengrad algorithm to improve its gradient calculation, accurately correcting the perspective difference between images with different focal lengths. On the FPGA, Gaussian filtering, Gaussian difference, extreme point detection, gradient calculation, and feature descriptor matching are designed as a parallel pipeline structure. The calculation formula for generating the scale space by convolution of the image with a multi-scale Gaussian kernel is as follows: Where I(x, y, σ) is the output image after convolution, I(x, y) is the input image, G(x, y, σ) is the Gaussian kernel, x and y are the image space coordinates, σ is the standard deviation, L(x, y, 2σ) is downsampled, and then convolved with filter kernels of different standard deviations. The operation is repeated to construct the complete scale space.

[0037] Specifically, Gaussian difference is used to detect feature points, and the calculation formula is: Where D G (x, y) is the Gaussian difference function, G σ1 and G σ2 are two Gaussian functions with different standard deviations, is the index term, The extreme points in the Gaussian difference result are taken as candidate feature points, and the accuracy and stability of the extreme points are guaranteed through interpolation and elimination.

[0038] In some embodiments, the gradient function is a Tenengrad gradient function, which uses templates in the 0° / 180° direction, 45° / 225° direction, 90° / 270° direction, and 135° / 315° direction to calculate the gradient distribution information in four directions of the area around the feature point.

[0039] Specifically, the Tenengrad function uses four unique directional templates—Sobel operator templates at 0° / 180°, 45° / 225°, 90° / 270°, and 135° / 315°—to perform a weighted calculation on the grayscale values of the pixels surrounding each feature point, comprehensively and accurately capturing the gradient changes in all directions of the image. The square root of the sum of the convolution results of each pixel with the four templates is then taken, and the calculated results for all pixels are combined to accurately reflect the gradient information of the feature point area. Furthermore, interpolation and filtering operations are used to further ensure the accuracy and stability of these feature points. For each feature point, the gradient direction distribution of the surrounding area is calculated, and the dominant direction is determined based on the distribution histogram. Each feature point is assigned a dominant direction, ensuring that the feature point remains unchanged despite rotational changes. This step ensures rotational invariance during image processing, ensuring stable matching of feature points even when the image angle changes.

[0040] Specifically, an image with a scale close to that of the feature point is selected in the scale space, and the gradient modulus and gradient angle of the pixels in the circular neighborhood of the feature point are calculated. The distribution histogram of the modulus value of the direction group is statistically analyzed, and the main direction and auxiliary direction of the feature point are determined by Gaussian weighting and quadratic interpolation fitting. The gradient angle calculation formula is: Where θ(x, y) is the angle at the coordinate point (x, y), L(x, y+1) and L(x, y-1) are the function values of the adjacent points in the y-axis direction, and L(x+1, y) and L(x-1, y) are the function values of the adjacent points in the x-axis direction.

[0041] Specifically, in order to improve the calculation speed and clarity evaluation results, the D′1 (0° / 180°), D′2 (45° / 225°), D′3 (90° / 270°), and D′4 (135° / 315°) templates are used instead. The improved grayscale gradient function is:

[0042]

[0043] Where, F tenengand is the improved grayscale gradient function, M and N are the height and width of the image, I(x, y) is the pixel value at the coordinate point (x, y), f(x, y) is the grayscale value of the pixel point (x, y), T otsu is the adaptive threshold, D′1, D′2, D′3, and D′4 are templates in four directions.

[0044] In some embodiments, the neighborhood area is divided into a number of neighborhood sub-areas, and the gradient information of the neighborhood sub-areas is calculated and concatenated to generate feature descriptors.

[0045] Specifically, an image with a scale close to that of the feature point is selected in scale space. The gradient modulus and gradient angle of the pixels within the circular neighborhood of the feature point are calculated. A histogram of the modulus distribution of the directional group is then statistically analyzed. The primary and secondary directions of the feature point are determined using Gaussian weighting and quadratic interpolation. The neighborhood is rotated according to the primary direction, and the gradient modulus and gradient angle are calculated and weighted. The neighborhood is then divided into 16 small regions. The gradient information for each region is calculated in detail, generating a 128-dimensional feature vector as a feature descriptor. This carefully designed descriptor efficiently captures local image features, providing strong support for accurate matching between images.

[0046] Optionally, the Manhattan distance is used to calculate the similarity between feature descriptors to perform feature point matching.

[0047] Specifically, based on the calculated feature descriptors, the Manhattan distance is used to quantify the matching degree of the feature point descriptors, and the feature point descriptor with the smallest Manhattan distance is selected as the matching result. By precisely matching the feature points between different images, the corresponding relationship between the images is determined. This precise matching enables perfect geometric alignment of images from different perspectives, effectively eliminating perspective differences and avoiding image misalignment and distortion caused by perspective deviation. After completing image matching and registration, the system seamlessly stitches the multi-perspective images together through advanced fusion techniques such as weighted averaging, generating a high-quality fused image. This process effectively eliminates seams between different images, ensuring good continuity and consistency in the output image. By employing the SIFT algorithm and fusing gradient functions, the present invention effectively addresses the registration challenges caused by camera perspective differences and processes image data in parallel on an FPGA platform, significantly improving processing speed and efficiency. The parallel execution of each step ensures efficient image processing, enabling the system to fully meet the stringent real-time processing and high-performance requirements of embedded applications such as autonomous driving. Leveraging the hardware acceleration advantages of FPGAs, the present invention not only significantly improves computing speed but also adapts well to the low power consumption and high stability requirements of in-vehicle environments.

[0048] In some embodiments, the step of stitching the registered images specifically includes: dividing the registered images into regions according to structural features and noise features of different regions, and selecting different fusion strategies for different regions of the image.

[0049] Specifically, by calculating the clarity feature value and selection metric for each region, the system selects the most appropriate fusion method based on the structural characteristics and noise of each region, thereby generating a higher-quality fused image. This dynamic scene-adaptive fusion method automatically adjusts the image integration strategy based on environmental changes such as lighting and obstacles, significantly reducing the problem of sub-image splicing distortion caused by block processing at image boundaries, ensuring that the autonomous driving system obtains optimal perception results in different scenarios.

[0050] Specifically, we first calculate the focus metric, considering focus as the accumulation of a focus and a noise signal: f i，j =G i，j +N i，j , where f i，j is the calculated value of the corresponding pixel according to the focus function, G i，j is the ideal focus signal, N i，j is the separated noise signal, N i，j It can be explained as image noise, which is the lack of structural feature information and the limitation of the focus operator. An ideal Gaussian function can be defined as follows: Where A is the maximum value of the Gaussian function, μ is the mean, and σ is the standard deviation. A, μ, and σ can be obtained by interpolation based on the independent variable z, and the Gaussian distribution is fitted by the least squares method to estimate the ideal focus signal G. i，j , the corresponding noise signal N i，j By calculating G i，j and N i，j The peak signal-to-noise ratio between the two is obtained: Where S(i, j) is the signal-to-noise ratio, f i，j is the calculated value of the corresponding pixel according to the focus function, N i，j To separate the noise signal, an entropy map that can represent the degree of defocus of the image sequence is obtained according to the above formula: Where σ is the standard deviation, G i，j is the ideal focus signal, and z is the independent variable.

[0051] Specifically, image regions are divided into three categories: low-noise regions with high structural significance, noise-interference regions with medium structural significance, and high-noise regions with low structural significance. Noise here primarily refers to the adverse effects of sample structural characteristics and optical 3D divergence. For example, smooth or transparent samples can affect the calculation of sharpness eigenvalues, and pixels with unclear structural features are easily affected by nearby clear pixels, blurring edge details in the fused image. Classification is based on the peak signal-to-noise ratio of the sharpness eigenvalues.

[0052] Specifically, based on the division of clarity feature values and the calculation of weighted coefficients, corresponding fusion rules are proposed, which are expressed as follows: in, Parameters a = 0.5 × maxH(i, j), b = 0.5 × a, where ψ(x, y) is the eigenvalue of the fused image at point (x, y), m is the index value with the largest weighted coefficient, ω k (x,y) is the weighted coefficient of the k-th layer image, I m (x, y) is the eigenvalue of the m-th layer image at point (x, y), Ik (x, y) is the eigenvalue of the k-th layer image at point (x, y), I n (x, y) is the eigenvalue of the n-th layer image at point (x, y), and H(i, j) is the clarity eigenvalue of the input image at point (i, j).

[0053] Specifically, in the low-noise area with high structural feature significance, the clarity eigenvalue is "sharp" in the Gaussian distribution, the peak signal-to-noise ratio is large, and the standard deviation is small; in the noise interference area with medium structural feature significance, the clarity eigenvalue is "bloated" in the Gaussian distribution, the peak signal-to-noise ratio is small, and the standard deviation is large; in the high-noise area with low structural feature significance, the clarity eigenvalue is "flat" in the Gaussian distribution, the peak signal-to-noise ratio is very small, and the standard deviation is large.

[0054] Specifically, for areas with clear structural features and low noise impact, the clearest image sequence is selected as the corresponding part of the fused image to retain the original information and speed up the fusion speed; for areas with clear structural features but affected by noise, local weighted averaging is used to avoid splicing "cracks" and reduce the impact of noise; for areas with little structural feature information or greatly affected by noise, the information of the nearby area is used to replace the current focus judgment based on the continuity of the sample surface.

[0055] In some embodiments, the step of dividing the registered image into regions based on the structural features and noise features of different regions specifically includes: calculating the clarity characteristic value of each feature point in the registered image and comparing it with a preset threshold, dividing the image into a high structural feature significant low noise area, a medium structural feature significant noise interference area and a low structural feature significant high noise area, wherein the clarity characteristic value includes the peak signal-to-noise ratio and the standard deviation.

[0056] Specifically, high structural feature significance and low noise areas: the image structural features in this type of area are rich in details and can clearly show the contours, structures and other features of the object, while the noise has little effect on them. The noise here mainly covers two aspects of adverse effects: on the one hand, there is the interference caused by the structural characteristics of the sample itself. For example, the smooth sample surface lacks sufficient structural feature information, which makes it difficult to accurately capture the features when calculating the clarity feature value; on the other hand, there are problems caused by optical three-dimensional divergence. For example, transparent samples will change the propagation of light, which also affects the accurate calculation of the clarity feature value. In this ideal case, the clarity feature value presents a "sharp" shape on the Gaussian distribution, that is, the peak signal-to-noise ratio is large, indicating that the image signal intensity is high and the noise interference is small. At the same time, the standard deviation is small, which means that the data is less discrete and the image features are stable and concentrated.

[0057] Specifically, the noise interference area with significant structural features: the image in this area can show a certain degree of structural feature information, and the general outline and partial structure of the object can be identified, but the noise interference is more obvious. The source of the noise is also the above-mentioned sample structural feature characteristics and optical three-dimensional divergence problems. In this case, pixels with unclear structural features are easily affected by nearby clear pixels, which leads to blurred edge details of the fused image during the image integration process. At this time, the clarity feature value appears in a "bloated" state on the Gaussian distribution, that is, the peak signal-to-noise ratio is small, indicating that the image signal is interfered with by noise and the intensity is reduced. At the same time, the standard deviation is large, indicating that the data is highly discrete, the image features are relatively scattered, and the stability is poor.

[0058] Specifically, in areas of low structural feature significance and high noise, images in these regions are nearly incapable of obtaining effective structural feature information or are severely interfered with by noise, rendering image features difficult to discern. The adverse effects of noise are further amplified in these areas, with smooth or transparent samples interfering more significantly with the calculation of sharpness eigenvalues. The sharpness eigenvalues exhibit a "flat" Gaussian distribution, with a low peak signal-to-noise ratio, indicating a weak image signal that is almost completely obscured by noise. A large standard deviation indicates extreme data dispersion and chaotic, disordered image features.

[0059] Specifically, classification is primarily based on the peak signal-to-noise ratio (PSNR) of the sharpness feature value. As an important indicator of image quality, PSNR intuitively reflects the signal-to-noise ratio in an image. By analyzing the PSNR, the image's focus can be accurately determined, leading to appropriate classification. The sharpness feature value is then calculated based on the calculated feature points and their main directions. In the FPGA, hardware multipliers and adders are used to calculate the sharpness feature value for each feature point according to a preset sharpness feature value calculation formula. The calculated sharpness feature value is then compared with a preset threshold. The FPGA's comparator and logic judgment unit classify the image region based on the PSNR. Regions with a high PSNR and a low standard deviation are classified as having clear structural features and low noise impact. Regions with a low PSNR and a large standard deviation are classified as having relatively clear structural features but affected by noise. Regions with a low PSNR and a large standard deviation are classified as having less structural information or being significantly affected by noise.

[0060] In some embodiments, the step of selecting different fusion strategies for different areas of the image specifically includes: for areas with high structural features and significant low noise, selecting the clearest image sequence as the corresponding part of the fused image; for areas with medium structural features and significant noise interference, using a local weighted averaging method to perform weighted averaging processing on pixels in the local area of the image; for areas with low structural features and significant high noise, using neighboring area information to replace current area information.

[0061] Specifically, for regions with high structural significance and low noise, the clearest image sequence is selected to serve as the corresponding portion of the fused image. This approach maximizes the preservation of information from the original image, as the clearest image sequence contains the richest and most accurate structural features and details. It also accelerates fusion, reduces unnecessary computation, and improves image processing efficiency.

[0062] Specifically, for areas with significant noise interference in structural features, a local weighted averaging method is used. By weighted averaging the pixels within a local area of the image, "cracks" can be effectively avoided during the stitching process. This weighted averaging process also reduces the impact of noise to a certain extent, making the fused image smoother and more natural, improving the overall image quality.

[0063] Specifically, for areas with low structural feature significance and high noise, information from nearby areas is used to replace the current focus judgment based on the continuity of the sample surface. Because this area lacks inherent structural feature information or is heavily affected by noise, accurate focus judgments are difficult to make, while nearby areas may have relatively stable features. By referencing information from nearby areas, the missing information in the current area can be filled, ensuring continuity and consistency in the overall fused image, avoiding noticeable flaws or anomalies.

[0064] These regional classification and fusion rules can flexibly adjust the image integration strategy according to the actual characteristics of different regions, thereby achieving higher quality and more efficient image integration effects, and providing a reliable data foundation for subsequent image processing and analysis.

[0065] In some embodiments, the step of stitching the registered images further includes: using gradient templates in four different directions to capture gradient information of the image, performing convolution operations on each pixel in the image with the gradient templates in four directions in turn, performing square operations and square root operations on the convolution results in each direction, screening out the calculation results that meet the preset gradient threshold and accumulating them to generate an image clarity evaluation function value; using the OTSU adaptive threshold algorithm to count the frequency of each gray level in the image, and then calculating the probability and mean of the foreground and background, as well as the mean of the entire image, by calculating the inter-class variance, finding the gray level that maximizes the inter-class variance among different gray levels as the gray level threshold, and dividing the image into foreground and background through the gray level threshold.

[0066] Specifically, multi-directional gradient templates are used, including templates in the 0° / 180° direction, 45° / 225° direction, 90° / 270° direction, and 135° / 315° direction. These templates can more comprehensively capture the gradient information of the image.

[0067] Next, for each pixel in the image, convolution is performed with the template in each of the four directions. The square root of the sum of the convolution results in each direction is taken. Only when the result meets certain conditions is the result for that pixel included in the final accumulation calculation. This operation is repeated for all pixels in the image, and the final cumulative value is obtained as the image clarity evaluation function value.

[0068] Using the OTSU adaptive thresholding algorithm, the image is divided into foreground and background based on the grayscale threshold. During this process, it is necessary to count the frequency of each grayscale in the image, and then calculate the probability and mean of the foreground and background, as well as the mean of the entire image.

[0069] A specific calculation method is then used to determine a value reflecting the difference between the foreground and background. To simplify the calculation, the formula is modified. During hardware programming, shifting or truncation operations are used to reduce the data bit width and speed up the calculation. The adaptive threshold is then determined by searching through different grayscale levels to find the grayscale that maximizes the value.

[0070] Finally, the image pixels are classified according to the determined adaptive threshold. Pixels with grayscale values less than the threshold are classified as background, and pixels with grayscale values greater than or equal to the threshold are classified as foreground. This improves the accuracy and sensitivity of subsequent edge gradient calculations.

[0071] Specifically, the OTSU adaptive threshold algorithm can be used to separate irrelevant pixels and enhance the accuracy and sensitivity of edge gradient-based sharpness feature calculations. The adaptive threshold Totsu is calculated using the OTSU algorithm, where the grayscale level that maximizes the inter-class variance between the foreground and background classes is used as the threshold. A 256-grayscale image is divided into two classes, foreground a and background b, using the grayscale threshold T. The calculation formula is: Where, σ 2 is the inter-class difference, ω a and ω b is the foreground gray level surrounding probability μ a and μ b represents the mean value of the foreground and background, and μ is the mean value of the entire image.

[0072] On the other hand, the present invention also provides a real-time multi-focus image fusion system based on FPGA, including: an acquisition module 10, used to acquire binocular images of different focal lengths and preprocess the binocular images; a feature point extraction module 20, used to extract feature points of the binocular images through a scale-invariant feature transformation algorithm; a gradient function module 30, used to calculate the gradient direction distribution information of the area around the feature point through a gradient function, and determine the main direction of the feature point; a descriptor generation module 40, used to calculate the gradient modulus and gradient direction in the neighborhood of each feature point, and rotate the neighborhood area according to the main direction of the feature point to generate a descriptor; a matching relationship determination module 50, used to calculate the similarity between feature descriptors through Manhattan distance to perform feature point matching, and then determine the matching relationship of the image; a fusion module 60, used to splice the aligned images to obtain a fused image.

[0073] Specifically, the FPGA hardware's parallel processing capabilities enable simultaneous processing of multiple data streams. When processing 1920×1080 resolution images, the hundreds of milliseconds of latency incurred by traditional CPUs or GPUs can be reduced to less than ten milliseconds, meeting the stringent real-time processing requirements of autonomous driving systems. This allows the system to quickly respond to image information, improving autonomous driving safety and reliability.

[0074] On the one hand, the image processing process is meticulously broken down into a five-stage, non-blocking pipeline: input, Gaussian filtering, scale-invariant feature transformation extraction, geometric registration, dynamic scene adaptation and fusion, and output. Each stage is equipped with independent computing units and register files to ensure continuous, uninterrupted data flow, improving the overall smoothness and efficiency of image processing. On the other hand, a state machine monitors the load at each stage in real time, dynamically adjusting the pipeline pause period based on the load, effectively avoiding global blocking caused by local computational delays and making the entire processing flow more stable and efficient.

[0075] Eight-bit unsigned integers are used to store gradient strengths, and a table lookup method is used instead of a floating-point function for direction quantization. Segmented gradient values with a step size of 1° over 360° are pre-calculated, and angles can be obtained by direct indexing when needed, simplifying the calculation process and improving efficiency. The differential Gaussian kernel is directly used for calculations to reduce repeated Gaussian filtering operations. The DOG values are truncated to a range of 0 to 255 for storage, effectively saving memory resources and improving storage and computational efficiency. In the DOG image, a 3×3 sliding window is used for neighborhood comparison, retaining only non-edge points, that is, points with a gradient amplitude difference greater than 15%. This avoids duplicate features, reduces the amount of calculation, and makes feature extraction more accurate and efficient.

[0076] The Manhattan distance accelerator also utilizes logic gates to implement absolute value operations, replacing square root units. This reduces Manhattan distance calculation resource consumption by 80%, significantly reducing hardware resource usage. A threshold is dynamically adjusted based on the median distance of the current matching pair, equal to 0.75 multiplied by the median, to improve matching accuracy. A circular buffer is used to store feature point queues, processing one pair of matches per cycle. Peak performance reaches 800,000 matches per second on specific FPGA models, significantly improving matching speed.

[0077] The OTSU threshold calculation unit is optimized by dividing the image into 16-by-16 pixel blocks and calculating the grayscale histogram for each block, reducing global traversal time and increasing calculation speed. By caching the data for each histogram block in a state machine and dynamically updating the global threshold, the single-frame threshold calculation time is significantly reduced from 8 milliseconds to 1.5 milliseconds, significantly improving threshold calculation efficiency.

[0078] Two memory modules with specific specifications are designed with a data flow duplex mode, allowing them to alternately cache image row data, achieving zero-wait pipelined read and write operations, improving data storage and access speeds and ensuring smooth data transmission. Utilizing the burst transfer mechanism of a specific interface, multiple data packets can be transmitted simultaneously, such as 64 pixels by 4 channels, reducing interface handshake overhead. Setting the burst length to 16 and the data width to 32 increases bandwidth to 240 megabytes per second, further improving data transmission efficiency.

[0079] Through the aforementioned technical approaches, this invention significantly improves image processing speed and efficiency on an FPGA platform, reduces computational effort, and optimizes storage and bandwidth. This not only meets the low power consumption and high stability requirements of in-vehicle environments, but also fulfills the stringent real-time processing and high performance demands of embedded applications like autonomous driving, providing reliable technical support for practical applications.

[0080] Clock gating technology can intelligently and dynamically shut down idle circuits based on the real-time operating status of each module. For example, using the TSIFT algorithm, which combines the SIFT algorithm with the Tenengrad gradient function, in image processing, once feature point screening is complete, the system automatically recognizes that the TSIFT computation unit is temporarily idle and immediately shuts down the clock signal to that unit. This prevents unnecessary computations, thus avoiding energy waste. Actual testing has verified that clock gating reduces overall power consumption by 20%. This technology not only reduces power consumption but also reduces the heat generated by unnecessary computations, improving system stability and reliability.

[0081] According to the different requirements of different modules for performance and power consumption, the present invention implements a precise voltage level adjustment strategy. For modules such as pipeline registers that have strict requirements for low latency, a relatively low voltage of 1.0V is used for power supply. This is because the pipeline registers are mainly responsible for the rapid transmission and temporary storage of data. The lower voltage can effectively reduce power consumption while meeting its low latency requirements. For high computing density modules, such as digital signal processing slices DSPSlice, since they undertake a large number of complex digital signal processing tasks and require higher computing speeds and processing capabilities, a power supply voltage of 1.2V is maintained to ensure their high-performance operation. Through this differentiated voltage level adjustment, a 15% reduction in overall power consumption is achieved without affecting the overall performance of the system. This refined voltage management strategy enables the system to maintain the best power consumption performance balance under different workloads.

[0082] Through the coordinated application of the above two power consumption optimization technologies, the present invention realizes efficient power consumption management on the FPGA platform, providing solid technical support for the stable and reliable operation of vehicle-mounted embedded systems such as autonomous driving.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0084] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0085] The above are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time multi-focus image fusion method based on FPGA, characterized in that the steps include: Collecting binocular images with different focal lengths and preprocessing the binocular images; Extracting feature points of the binocular image by using a scale-invariant feature transformation algorithm; Calculate the gradient direction distribution information of the area around the feature point by using a gradient function to determine the main direction of the feature point; Calculating the gradient modulus and gradient direction in the neighborhood of each feature point, and rotating the neighborhood area according to the main direction of the feature point to generate a descriptor; The similarity between feature descriptors is calculated by Manhattan distance to match feature points and determine the matching relationship of images; The registered images are stitched together to obtain a fused image.

2. The real-time multi-focus image fusion method based on FPGA according to claim 1, characterized in that: The pre-processing steps include: The binocular image is smoothed by a Gaussian filter to remove noise.

3. The real-time multi-focus image fusion method based on FPGA according to claim 1, characterized in that: The step of extracting the feature points of the binocular image by using a scale-invariant feature transformation algorithm specifically includes: Filtering the binocular image using a multi-scale Gaussian kernel to generate a multi-scale image set and then construct a scale space; Images of adjacent scales in the scale space are differentiated to obtain a Gaussian difference image, and local extreme value points in the Gaussian difference image are selected as feature points.

4. The real-time multi-focus image fusion method based on FPGA according to claim 1, characterized in that: The gradient function is a Tenengrad gradient function, which uses templates in the 0° / 180° direction, 45° / 225° direction, 90° / 270° direction, and 135° / 315° direction to calculate the gradient distribution information in four directions of the area around the feature point.

5. The real-time multi-focus image fusion method based on FPGA according to claim 1, characterized in that: The step of generating a descriptor specifically includes: The neighborhood area is divided into a plurality of neighborhood sub-areas, and the gradient information of the neighborhood sub-areas is calculated and spliced to generate feature descriptors.

6. The real-time multi-focus image fusion method based on FPGA according to claim 1, characterized in that: The step of stitching the registered images specifically includes: The registered image is divided into regions according to the structural and noise characteristics of different regions, and different fusion strategies are selected for different regions of the image.

7. The FPGA-based real-time multi-focus image fusion method according to claim 6, characterized in that: The step of dividing the registered image into regions according to the structural features and noise features of different regions specifically includes: The clarity characteristic value of each feature point in the registered image is calculated and compared with a preset threshold, and the image is divided into a high structural feature significant low noise area, a medium structural feature significant noise interference area and a low structural feature significant high noise area, wherein the clarity characteristic value includes the peak signal-to-noise ratio and the standard deviation.

8. The FPGA-based real-time multi-focus image fusion method according to claim 7, characterized in that: The step of selecting different fusion strategies for different areas of the image specifically includes: For areas with high structural features and significant low noise, the clearest image sequence is selected as the corresponding part of the fused image; For the noise interference area with significant structural features, the local weighted average method is used to perform weighted average processing on the pixels in the local area of the image; For areas with low structural features and significant high noise, the information of the neighboring areas is used to replace the current area information.

9. The real-time multi-focus image fusion method based on FPGA according to claim 1, characterized in that: The step of stitching the registered images further comprises: Four gradient templates in different directions are used to capture the gradient information of the image. Each pixel in the image is convolved with the gradient templates in four directions in turn. The convolution results in each direction are squared and squared, and the calculation results that meet the preset gradient threshold are screened and accumulated to generate the image clarity evaluation function value. The OTSU adaptive threshold algorithm is used to count the frequency of each gray level in the image, and then the probability and mean of the foreground and background, as well as the mean of the entire image, are calculated. By calculating the inter-class variance, the gray level that maximizes the inter-class variance is found among different gray levels as the gray level threshold. The image is divided into foreground and background using the gray level threshold.

10. A real-time multi-focus image fusion system based on FPGA, characterized in that: include: An acquisition module, used for acquiring binocular images of different focal lengths and preprocessing the binocular images; A feature point extraction module is used to extract feature points of the binocular image through a scale-invariant feature transformation algorithm; A gradient function module is used to calculate the gradient direction distribution information of the area around the feature point through a gradient function to determine the main direction of the feature point; A descriptor generation module is used to calculate the gradient modulus and gradient direction in the neighborhood of each feature point, and rotate the neighborhood area according to the main direction of the feature point to generate a descriptor; A matching relationship determination module is used to calculate the similarity between feature descriptors through Manhattan distance to match feature points and then determine the matching relationship of images; The fusion module is used to stitch the registered images to obtain a fused image.

Images