An FPGA-based image stitching method, system, medium and device

Abstract

The present invention discloses an image stitching method, system, medium and device based on FPGA, which relates to the field of computer vision and includes: obtaining multiple images to be stitched from an image sensor, and analyzing and recording the image features of each image; calculating the complexity of the image according to the image features, evaluating the power consumption required for processing the image according to the complexity of the image, and determining an initial processing strategy accordingly; dynamically adjusting the processing strategy according to the power consumption and the image complexity; extracting key feature points from each image to generate feature descriptors; matching the generated feature descriptors, recording the matching point pairs, and calculating the transformation parameters between the images according to the matching results; performing geometric transformation and fusion on each image for the calculated transformation parameters to generate a final stitched image. The present invention enhances the accuracy of geometric transformation and provides richer data support by using an RGB-D camera to obtain the color and depth information of the image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image stitching method, system, medium, and device based on FPGA. Background Art

[0002] As an important part of computer vision, image stitching technology has developed rapidly in recent years with the progress of hardware computing power and algorithms. Traditional methods rely on CPU processing to achieve seamless fusion of multiple images through feature point detection, descriptor generation, matching, and geometric transformation. However, in the face of application scenarios with high real-time requirements for video streams, the CPU method shows problems of large computational resource consumption and long processing time when dealing with large-scale data sets or high-resolution images. Although the GPU significantly improves the processing speed with its powerful parallel computing ability, its high power consumption limits its application in embedded systems and mobile devices, and the original architecture design of the GPU is not aimed at image stitching tasks, resulting in limitations in flexibility and customization. In contrast, FPGA provides a better performance-to-power ratio with its highly flexible and efficient characteristics and allows custom logic circuits according to specific requirements, making it very suitable for applications that require a balance of high performance and low power consumption.

[0003] Although existing image stitching technologies can meet basic requirements, they still face challenges when dealing with complex images. Traditional stitching methods adopt fixed processing strategies and cannot dynamically adjust computational accuracy and resource allocation, resulting in waste of resources in simple image processing, while in complex image processing, the quality may be affected due to insufficient resources. In addition, most solutions fail to fully utilize the color and depth information provided by RGB-D cameras, especially the application value in geometric transformation is ignored. Existing algorithms pay more attention to the quality of the final result and less consider power consumption management throughout the process, which is particularly important in resource-constrained environments. The present invention solves the above problems through an FPGA-based method, combines dynamic processing strategy adjustment, makes full use of RGB-D information, and efficient image fusion technology, achieving efficient and low-power image stitching, which is particularly suitable for occasions of real-time processing of high-resolution images. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an FPGA-based image stitching method to solve the problem that traditional stitching methods adopt fixed processing strategies and cannot dynamically adjust computational accuracy and resource allocation, resulting in waste of resources in simple image processing, while in complex image processing, the quality may be affected due to insufficient resources.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an FPGA-based image stitching method, which includes:

[0008] Obtain multiple images to be stitched from an image sensor, and analyze and record the image features of each image;

[0009] Calculate the complexity of the image according to the image features, evaluate the power consumption required to process the image according to the complexity of the image, and accordingly determine the initial processing strategy;

[0010] Dynamically adjust the processing strategy according to the power consumption and image complexity;

[0011] Extract key feature points from each image to generate feature descriptors;

[0012] Match the generated feature descriptors, record the matching point pairs, and calculate the transformation parameters between the images according to the matching results;

[0013] Perform geometric transformation and fusion on each image for the calculated transformation parameters to generate the final stitched image.

[0014] As a preferred solution of the FPGA-based image stitching method of the present invention, wherein: the step of obtaining multiple images to be stitched from an image sensor and analyzing and recording the image features of each image is specifically:

[0015] Use an RGB-D camera to collect multiple images to be stitched;

[0016] Calculate the average brightness and contrast of each image. The formula for calculating the average brightness of each image is:

[0017]

[0018] wherein, L k represents the average brightness of the kth image, M represents the total number of pixels, and Ω k represents the set of all pixel points of the kth image, and R(x, y), G(x, y), and B(x, y) respectively represent the red, green, and blue component values at the pixel point (x, y).

[0019] As a preferred solution of the FPGA-based image stitching method of the present invention, wherein: the step of calculating the complexity of the image according to the image features, evaluating the power consumption required to process the image according to the complexity of the image, and accordingly determining the initial processing strategy is specifically:

[0020] Calculate the complexity D k of the image according to the average brightness and contrast of each image. The expression is:

[0021] D k = α·C k+β·(σ L / L k );

[0022] Among them, D k represents the complexity of the k-th image, indicating the richness of image features, α represents the weight coefficient of contrast, β represents the weight coefficient of change rate, and σ L represents the standard deviation of the average brightness of the k-th image;

[0023] According to the complexity D k of the image, the images are classified into three categories: simple images, medium-complexity images, and complex images;

[0024] Set the complex thresholds T1 and T2;

[0025] When D k < T1, it is a simple image, indicating fewer image features, uniform brightness, and low contrast, and a low-precision calculation strategy is adopted;

[0026] When T1 ≤ D k < T2, it is a medium-complexity image, indicating that the image has certain feature information, moderate brightness and texture changes, and a medium-precision calculation strategy is adopted;

[0027] When D k ≥ T2, it is a complex image, indicating rich image features, large brightness changes, and high contrast, and a high-precision calculation strategy is suitable;

[0028] According to the complexity D k of the image, calculate the power consumption P k required to process the image, and the expression is:

[0029]

[0030] Among them, P k represents the power consumption required to process the k-th image, P base represents the basic power consumption during operation, and η represents the weight coefficient of complexity.

[0031] As a preferred solution of the FPGA-based image stitching method described in the present invention, wherein: the processing strategy is dynamically adjusted according to the power consumption and image complexity, and the specific steps are as follows:

[0032] For simple images, when it is monitored that the power consumption P k is significantly lower than the set threshold and the FPGA resource utilization rate has not reached the upper limit, switch from the original fast feature detection algorithm to the SURF algorithm;

[0033] By applying multi-scale detection to increase the number of matching points, thereby improving the accuracy of stitching;

[0034] Estimate the homography matrix using RANSAC combined with the Levenberg-Marquardt local optimization algorithm to obtain a higher-quality stitching result;

[0035] For medium-complexity images, maintain the initially selected calculation strategy unchanged while real-time tracking the power consumption changes; once the power consumption exceeds the preset limit, trigger an alarm and automatically reduce the number of feature points or simplify the matching process;

[0036] Dynamically adjust the resolution of the input image according to actual needs;

[0037] For complex images, when the power consumption exceeds the budget, simplify the originally used algorithm SURF for generating complex feature descriptors to a lighter-weight ORB to reduce the computational burden;

[0038] Perform feature point detection only at the edges and corners of the image, ignoring flat areas to reduce the overall computational amount;

[0039] Divide the large-size image into small sub-blocks, process them in parallel on different logic units of the FPGA, and then summarize the results to complete the final stitching task;

[0040] Adopt a multi-layer fusion technology to generate a preliminary stitching result, and then refine the boundaries and overlapping areas.

[0041] As a preferred solution of the FPGA-based image stitching method described in the present invention, wherein: extracting key feature points from each image and generating feature descriptors, the specific steps are as follows:

[0042] Adopt the Speeded Up Robust Features (SURF) algorithm to detect key points in the image;

[0043] For each detected key point, divide the area around each key point into multiple sub-regions, calculate the gradient direction histogram mHoG in each sub-region, and combine it with the depth difference histogram DDH;

[0044] Connect the gradient direction histograms and depth difference histograms of all sub-regions to form a final feature descriptor vector;

[0045] Normalize the feature descriptor vector to obtain the normalized feature descriptor strength F(r), and the expression is:

[0046]

[0047] Among them, F(r) represents the feature descriptor strength of the r-th key point after normalization, w i represents the weight factor of the i-th sub-region, h i represents the gradient direction histogram value of the i-th sub-region, b iThe value of the depth difference histogram representing the i-th sub-region, and n represents the total number of divided sub-regions.

[0048] As a preferred solution of the FPGA-based image stitching method described in the present invention, wherein: matching the generated feature descriptors, recording the matching point pairs, and calculating the transformation parameters between the images according to the matching results. The specific steps are as follows:

[0049] Perform feature point matching using the bidirectional nearest neighbor (BNN) algorithm based on the extracted key feature points and their descriptors;

[0050] Construct a scoring function based on the key feature points and their descriptors to evaluate the reliability of each pair of matching points. The expression is:

[0051]

[0052] where S(p δ , q j ) represents the reliability score value of the matching point pair p δ and q j . p δ represents the δ-th key point extracted from the first image, q j represents the j-th key point extracted from the second image, d(p δ , q j ) represents the Euclidean distance between the two descriptors of p δ and q j . λ represents the parameter controlling the slope of the sigmoid function, μ represents the average distance, ΔR(p δ , q j ) represents the depth difference between p δ and q j . γ represents the weight factor, and σ R represents the standard deviation of the depth differences of all matching point pairs;

[0053] Subsequently, for the reliable matching point pairs screened by the scoring function, use the random sample consensus (RANSAC) algorithm to estimate the geometric transformation parameters between the images and obtain the homography matrix H.

[0054] As a preferred solution of the FPGA-based image stitching method described in the present invention, wherein: performing geometric transformation and fusion on each image for the calculated transformation parameters to generate the final stitched image. The specific steps are as follows:

[0055] For each image to be stitched, perform an affine transformation on it using the homography matrix H, and apply this homography matrix to each image to be stitched so that all images can be in the same coordinate system;

[0056] Perform an affine transformation on each image using an FPGA for stitching to obtain a stitched image;

[0057] Adopt a weighted average fusion method to mix the overlapping regions in the stitched image, eliminate the stitching traces, and generate the final stitched image.

[0058] In a second aspect, the present invention provides an FPGA-based image stitching system, including: an acquisition module, an adjustment module, a descriptor module, a matching module, and a fusion module:

[0059] The acquisition module is responsible for obtaining multiple images to be stitched from an image sensor, analyzing and recording the color information RGB and depth information Depth of each image, and calculating the average brightness and contrast of the image;

[0060] The adjustment module is used to calculate the image complexity based on image features, evaluate the power consumption required to process these images according to the complexity, and at the same time, dynamically adjust the processing strategy;

[0061] The descriptor module uses methods such as the SURF algorithm to detect key feature points in the image and generate corresponding feature descriptors;

[0062] The matching module uses the bidirectional nearest neighbor (BNN) algorithm for feature point matching and estimates the geometric transformation parameters between images based on the matching results using the RANSAC algorithm;

[0063] The fusion module is responsible for using the FPGA to accelerate the execution of affine transformation or other geometric transformation operations to convert all images into the same coordinate system, perform weighted average fusion on the overlapping regions, eliminate the stitching traces, and generate a seamless and visually coherent final stitched image.

[0064] In a third aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and: when the computer program is executed by the processor, any step of the FPGA-based image stitching method described in the first aspect of the present invention is implemented.

[0065] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and: when the computer program is executed by the processor, any step of the FPGA-based image stitching method described in the first aspect of the present invention is implemented.

[0066] The beneficial effects of the present invention are as follows: By using an RGB-D camera to obtain the color and depth information of the image, the accuracy of geometric transformation is enhanced, providing richer data support. Secondly, by calculating the image complexity and evaluating the power consumption, the optimal allocation of resources is achieved, ensuring that simple images do not waste resources and complex images are fully processed, thereby improving the overall efficiency of the system. Dynamically adjusting the processing strategy further optimizes the balance between real-time performance and energy efficiency, and is particularly suitable for resource-constrained application scenarios. Using the SURF algorithm to extract key feature points not only speeds up the detection speed but also improves the stability and reliability of matching. The combination of the two-way nearest neighbor BNN algorithm and the scoring function reduces the mis-matching phenomenon and ensures the accuracy of matching; the RANSAC algorithm enhances the robustness of the transformation parameter estimation. Finally, using FPGA hardware acceleration for affine transformation and processing overlapping regions through a weighted average fusion method, a seamless and visually coherent high-quality stitched image is achieved. In summary, the present invention greatly improves the speed and quality of image stitching while ensuring high precision and low power consumption, and is applicable to occasions where high-resolution images are processed in real time. Description of the Drawings

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0068] Figure 1 It is a flowchart of the FPGA-based image stitching method in Embodiment 1.

[0069] Figure 2 It is a schematic diagram of the FPGA-based image stitching system in Embodiment 1. Detailed Embodiments

[0070] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following will give a detailed description of the specific embodiments of the present invention with reference to the drawings in the specification.

[0071] In the following description, many specific details are set forth to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0072] Second, the "one embodiment" or "embodiment" referred to herein means a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The appearances of "in one embodiment" in different places in this specification do not all refer to the same embodiment, nor are they separate or alternative embodiments that are mutually exclusive of other embodiments.

[0073] Embodiment 1, referring to Figure 1 and Figure 2 , is the first embodiment of the present invention. This embodiment provides an FPGA-based image stitching method, including the following steps:

[0074] S1. Obtain multiple images to be stitched from an image sensor and analyze and record the image features of each image;

[0075] Use an RGB-D camera (Intel RealSense) to collect multiple images to be stitched. The reason for choosing an RGB-D camera here is that it can simultaneously obtain the color information (RGB) and depth information (Depth) of the image, providing richer data support for subsequent geometric transformations;

[0076] The image features of each image include the average brightness and contrast of each image;

[0077] Calculate the average brightness and contrast of each image. The formula for calculating the average brightness of each image is:

[0078]

[0079] where L k represents the average brightness of the kth image, M represents the total number of pixels, and Ω k represents the set of all pixel points of the kth image. R(x, y), G(x, y), and B(x, y) respectively represent the red, green, and blue component values at the pixel point (x, y), with a value range of [0, 255]. 0.299, 0.587, and 0.114 are the weight coefficients of the red, green, and blue components respectively;

[0080] In step S1, using an RGB-D camera (Intel RealSense) to obtain multiple images to be stitched and analyze and record the image features is an innovation point. The depth information provided by the RGB-D camera provides richer data support for subsequent geometric transformations, which has obvious advantages compared with traditional cameras that only provide two-dimensional information. Specifically, the depth information can better understand the spatial structure of the scene, thereby improving the accuracy and quality of image stitching. This technology is particularly important in fields such as 3D reconstruction and augmented reality.

[0081] S2. Calculate the complexity of the image based on the image features, evaluate the power consumption required for processing the image according to the complexity of the image, and determine the initial processing strategy accordingly;

[0082] Calculate the complexity D of the image based on the average brightness and contrast of each image k , and the expression is:

[0083] D k = α·C k + β·(σ L / L k );

[0084] Among them, D k represents the complexity of the k-th image, represents the richness of image features, α represents the weight coefficient of contrast, β represents the weight coefficient of the change rate, and σ L represents the standard deviation of the average brightness of the k-th image;

[0085] Classify the images into three categories: simple images, medium-complexity images, and complex images according to the complexity D of the image k ;

[0086] Set the complex thresholds T1 and T2;

[0087] When D k < T1, it is a simple image, indicating that the image features are few, the brightness is uniform, and the contrast is low, and a low-precision calculation strategy is adopted;

[0088] When T1 ≤ D k < T2, it is a medium-complexity image, indicating that the image has certain feature information, and the brightness and texture changes are moderate, and a medium-precision calculation strategy is adopted;

[0089] When D k ≥ T2, it is a complex image, indicating that the image features are rich, the brightness changes greatly, and the contrast is high, and a high-precision calculation strategy is suitable;

[0090] Calculate the power consumption P required for processing the image according to the complexity D of the image k ; k ;

[0091]

[0092] Among them, P k represents the power consumption required for processing the k-th image, and P base represents the basic power consumption during operation, and η represents the weight coefficient of complexity.

[0093] S3. Dynamically adjust the processing strategy according to the power consumption and image complexity;

[0094] For simple images, when the monitored power consumption Pk When it is significantly lower than the set threshold and the FPGA resource utilization rate does not reach the upper limit, switch from the original fast feature detection algorithm to the SURF algorithm;

[0095] By applying multi-scale detection to increase the number of matching points, thereby improving the accuracy of stitching;

[0096] Use RANSAC combined with the Levenberg-Marquardt of the local optimization algorithm to estimate the homography matrix and obtain a higher-quality stitching result;

[0097] For medium-complexity images, maintain the initially selected calculation strategy unchanged and simultaneously track the power consumption change in real time; once the power consumption exceeds the preset limit, trigger an alarm and automatically reduce the number of feature points or simplify the matching process;

[0098] Dynamically adjust the resolution of the input image according to actual needs;

[0099] For complex images, when the power consumption exceeds the budget, simplify the originally used SURF algorithm for generating complex feature descriptors to a lighter-weight ORB to reduce the computational burden;

[0100] Perform feature point detection only at the edges and corners of the image, ignoring flat areas, to reduce the overall computational amount;

[0101] Divide the large-size image into small sub-blocks, process them in parallel on different logic units of the FPGA respectively, and then summarize the results to complete the final stitching task;

[0102] Adopt a multi-layer fusion technology to generate a preliminary stitching result and then refine the boundaries and overlapping areas.

[0103] The core of the present invention lies in dynamically adjusting the processing strategy according to the image complexity and power consumption. Traditional methods usually adopt a fixed processing strategy, which leads to resource waste or insufficient performance. The present invention can adaptively adjust the calculation accuracy and resource allocation according to different types of images by introducing an image complexity evaluation mechanism, thereby optimizing the overall performance;

[0104] It solves the problem in the prior art that the processing strategy cannot be flexibly adjusted according to the image characteristics. Specifically, when processing simple images, the accuracy is improved without increasing power consumption; for complex images, when the power consumption exceeds the budget, the overall performance is ensured by reducing the calculation accuracy, avoiding the decline of stitching quality due to insufficient resources.

[0105] S4. Extract key feature points from each image to generate feature descriptors;

[0106] The Speeded Up Robust Features (SURF) algorithm is used to detect key points in images. SURF is a fast and reliable feature point detector that locates stable key points by finding the extreme points of the Hessian matrix in spaces of different scales. SURF is chosen because it has good invariance to rotation and scale changes, and compared with the traditional Scale-Invariant Feature Transform (SIFT) algorithm, SURF is faster and more suitable for real-time processing requirements;

[0107] For each detected key point, the area around each key point is divided into multiple sub-regions. The multi-scale Histogram of Oriented Gradients (mHoG) is calculated within each sub-region and combined with the Depth Difference Histogram (DDH). Specifically:

[0108] Within each sub-region, the gradient magnitude and direction of each pixel are calculated by applying the Sobel operator. The gradient direction is quantized into discrete angular intervals, for example, every 20 degrees between 0 and 360 degrees as an interval. This can map continuous direction values into a finite number of discrete intervals. A histogram is created for each sub-region, where each bin corresponds to a quantized gradient direction interval. Then, all pixels within the sub-region are traversed, and their corresponding gradient magnitudes are accumulated into the corresponding bins according to their gradient directions. The finally formed histogram of oriented gradients reflects the gradient distribution in different directions within this sub-region;

[0109] The depth map of each image is obtained from the RGB-D sensor. A pair of adjacent sub-regions is selected, and the average depth between them is calculated. Then, the depth difference between the two sub-regions is obtained. This depth difference is taken as the difference between the two. A series of discrete depth difference intervals are set, and the number of samples falling within these intervals is counted, thereby constructing the Depth Difference Histogram (DDH);

[0110] Finally, the HoG and DDH of each sub-region are concatenated to form a comprehensive feature description. This comprehensive feature vector can simultaneously express local texture information (provided by HoG) and three-dimensional structure information (provided by DDH);

[0111] The histograms of oriented gradients and depth difference histograms of all sub-regions are concatenated to form the final feature descriptor vector;

[0112] The feature descriptor vector is normalized to obtain the normalized feature descriptor strength F(r), and the expression is:

[0113]

[0114] where F(r) represents the feature descriptor strength of the r-th key point after normalization, w i represents the weight factor of the i-th sub-region, h iIndicates the histogram of gradient directions value of the \(i\)-th sub-region, \(b\). i Indicates the histogram of depth differences value of the \(i\)-th sub-region, and \(n\) represents the total number of divided sub-regions;

[0115] By using the SURF algorithm for feature point detection and combining with the histogram of depth differences to generate feature descriptors, the present invention not only significantly improves the effect and efficiency of image stitching, but also solves the technical limitation problems existing in traditional methods. This method is particularly suitable for application scenarios that require high-quality image stitching, such as panoramic photography and virtual reality;

[0116] The intensity \(F(r)\) of the normalized feature descriptor is used to measure the significance of the descriptor and provides better stability and accuracy during the matching process.

[0117] S5. Match the generated feature descriptors, record the matching point pairs, and calculate the transformation parameters between the images according to the matching results;

[0118] Based on the extracted key feature points and their descriptors, the bidirectional nearest neighbor (BNN) algorithm is used for feature point matching. The traditional unidirectional nearest neighbor method may lead to incorrect matches, while BNN can more accurately screen out the true matching point pairs by checking the nearest neighbor relationships in both directions. The BNN is selected because it can reduce the possibility of false matches while maintaining the matching accuracy;

[0119] Based on the key feature points and their descriptors, a scoring function is constructed to evaluate the reliability of each pair of matching points. The expression is:

[0120]

[0121] where \(S(p\) δ , \(q\) j ) represents the reliability score value of the matching point pair \(p\) δ and \(q\) j . \(p\) δ represents the \(\delta\)-th key point extracted from the first image, \(q\) j represents the \(j\)-th key point extracted from the second image, \(d(p\) δ , \(q\) j ) represents the Euclidean distance between the descriptors of \(p\) δ and \(q\) j . \(\lambda\) represents the parameter controlling the slope of the sigmoid function, \(\mu\) represents the average distance, \(\Delta R(p\) δ , \(q\) j ) represents the depth difference between \(p\) δ and \(q\) j . \(\gamma\) represents the weight factor, \(\sigma\) R represents the standard deviation of the depth differences of all matching point pairs; \(S(p\) δ , \(q\) j)The value range is between [0, 1]. The larger the value, the higher the matching reliability, which helps to select the best matching point pairs from numerous candidate matches;

[0122] Subsequently, for the reliable matching point pairs filtered by the scoring function, the random sample consensus algorithm RANSAC is used to estimate the geometric transformation parameters between the images, obtaining the homography matrix H. The reason for choosing RANSAC is that it has good resistance to noise and outliers, ensuring that the finally obtained transformation parameters are accurate enough.

[0123] S6. Perform geometric transformation and fusion on each image for the calculated transformation parameters to generate the final stitched image;

[0124] For each image to be stitched, use the homography matrix H to perform an affine transformation on it, and apply this homography matrix to each image to be stitched so that all images can be in the same coordinate system. To improve efficiency and accuracy, we map these transformation operations to the FPGA hardware for execution, using its parallel processing ability to accelerate the calculation process;

[0125] Use the FPGA to perform affine transformation on each image for stitching to obtain the stitched image. These transformation operations will adjust the position, size, and angle of the image according to the homography matrix, enabling them to be seamlessly stitched together;

[0126] Adopt the weighted average fusion method to mix the overlapping areas in the stitched image, eliminate the stitching traces, and generate the final stitched image.

[0127] This embodiment also provides an FPGA-based image stitching system, including: an acquisition module, an adjustment module, a descriptor module, a matching module, and a fusion module:

[0128] The acquisition module is responsible for obtaining multiple images to be stitched from the image sensor, analyzing and recording the color information RGB and depth information Depth of each image, as well as calculating the average brightness and contrast of the image;

[0129] The adjustment module is used to calculate the image complexity according to the image features, evaluate the power consumption required to process these images according to the complexity, and at the same time, dynamically adjust the processing strategy;

[0130] The descriptor module uses methods such as the SURF algorithm to detect key feature points in the image and generate corresponding feature descriptors;

[0131] The matching module uses the bidirectional nearest neighbor (BNN) algorithm for feature point matching and estimates the geometric transformation parameters between the images based on the matching results using the RANSAC algorithm;

[0132] The fusion module is responsible for using FPGA to accelerate the execution of affine transformation or other geometric transformation operations to transform all images into the same coordinate system, perform weighted average fusion on the overlapping regions, eliminate the stitching marks, and generate a seamless and visually coherent final stitched image.

[0133] This embodiment also provides a computer device applicable to the case of the FPGA-based image stitching method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the FPGA-based image stitching method proposed in the above embodiment.

[0134] The computer device can be a terminal. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0135] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the FPGA-based image stitching method proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM for short), Electrically Erasable Programmable Read-Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM for short), Programmable Read-Only Memory (PROM for short), Read-Only Memory (ROM for short), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0136] In summary, the present invention enhances the accuracy of geometric transformation and provides richer data support by using an RGB-D camera to obtain the color and depth information of an image. Secondly, by calculating the image complexity and evaluating the power consumption, the optimal allocation of resources is achieved, ensuring that simple images do not waste resources and complex images are fully processed, thereby improving the overall efficiency of the system. Dynamically adjusting the processing strategy further optimizes the balance between real-time performance and energy efficiency, making it particularly suitable for resource-constrained application scenarios. The SURF algorithm is used to extract key feature points, which not only speeds up the detection speed but also improves the stability and reliability of matching. The combination of the two-way nearest neighbor BNN algorithm and the scoring function reduces the mis-matching phenomenon and ensures the accuracy of matching; the RANSAC algorithm enhances the robustness of the transformation parameter estimation. Finally, the affine transformation is accelerated using FPGA hardware, and the overlapping regions are processed by the weighted average fusion method to achieve seamless and visually coherent high-quality stitched images. In summary, the present invention significantly improves the speed and quality of image stitching while ensuring high precision and low power consumption, and is applicable to occasions of real-time processing of high-resolution images.

[0137] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the image stitching method based on FPGA are given.

[0138] In this example, a set of multi-frame high-resolution images collected by an Intel RealSense RGB-D camera are selected as test samples. These images cover different scene complexities, including simple indoor environments, moderately complex natural landscapes, and complex building structures, to comprehensively evaluate the effectiveness and superiority of the inventive method.

[0139] First, 10 images to be stitched are obtained through the RGB-D camera, and the color information (RGB) and depth information (Depth) of each image are recorded. To ensure the consistency and accuracy of the data, all images are taken under the same conditions, maintaining the same exposure settings and viewing angle changes. Then, feature analysis is performed on each image, and the average brightness and contrast are calculated. The two metrics are quantified using the formulas L k and C k respectively. This step not only provides the necessary data support for subsequent geometric transformation but also enables the system to dynamically adjust the processing strategy according to the specific characteristics of the image.

[0140] Next, according to the calculated average brightness and contrast, the complexity C k of the image is further calculated. By setting thresholds T1 and T2, the images are divided into three categories: simple, moderately complex, and complex. For each category of images, the power consumption P k, and based on this, the initial processing strategy was determined. During this process, special attention was paid to the effective utilization of resources to ensure the best performance and energy consumption balance in different types of image processing;

[0141] Subsequently, the SURF algorithm was used to extract key feature points from each image and generate feature descriptors. Compared with the traditional SIFT algorithm, SURF is not only faster but also has good invariance to rotation and scale changes, which is very suitable for real-time processing requirements. The area around each detected key point is divided into multiple sub-regions, and the gradient direction histogram mHoG is calculated within each sub-region, and combined with the depth difference histogram DDH to form the final descriptor vector. This step significantly improves the accuracy and reliability of feature point detection and lays a solid foundation for subsequent matching and transformation steps;

[0142] Next, the bidirectional nearest neighbor (BNN) algorithm was used for feature point matching, and a scoring function was constructed to evaluate the reliability of the matching point pairs. The BNN algorithm reduces the possibility of false matches by checking the nearest neighbor relationships in both directions, and the scoring function further ensures the accuracy of the matching. For the reliable matching point pairs after screening, the random sample consensus algorithm (RANSAC) was used to estimate the geometric transformation parameters between the images, and the homography matrix H was obtained. The RANSAC algorithm has good resistance to noise and outliers, ensuring that the finally obtained transformation parameters are accurate enough;

[0143] Finally, by applying the homography matrix H to each image to be stitched for affine transformation, all images were transformed into the same coordinate system, achieving seamless stitching. To improve efficiency and accuracy, these transformation operations were mapped to the FPGA hardware for execution, taking advantage of its parallel processing ability to accelerate the calculation process. In addition, a weighted average fusion method was used to blend the overlapping regions in the stitched images, eliminating the stitching artifacts and generating the final high-quality stitched image.

[0144] Specifically, it is shown in Table 1 below:

[0145] Table 1 Experimental data record table

[0146]

[0147]

[0148] It can be seen from the data in Table 1 that the present invention shows obvious advantages in many aspects:

[0149] Power consumption reduction: Compared with the prior art, the power consumption of the present invention is reduced by about 22%, that is, from 3.2 watts to 2.5 watts; this is mainly due to the ability to dynamically adjust the processing strategy according to the image complexity, ensuring the optimal allocation of resources and avoiding unnecessary energy consumption.

[0150] Increase in the number of feature points and matching point pairs: The number of feature points extracted by the present invention has increased by 25% (from 1,200 to 1,500), and the number of matching point pairs has increased by 50% (from 800 to 1,200); this indicates that the application of the SURF algorithm has improved the accuracy and reliability of feature point detection and enhanced the stability of subsequent matching.

[0151] Significant reduction in stitching time: The stitching time of the present invention is only half of that of the prior art, reduced from 12 seconds to 6 seconds; this is because the FPGA hardware has accelerated the affine transformation operation and optimized the entire processing flow, thus significantly improving the processing speed.

[0152] Improvement in the quality of the stitched image: The stitched image generated by the present invention has higher quality and better visual effects; this benefits from the application of the weighted average fusion method, which ensures a smooth transition in the overlapping area and eliminates obvious stitching traces.

[0153] Increase in energy efficiency ratio: The energy efficiency ratio of the present invention reaches 0.48 W / frame, much higher than 0.27 W / frame of the prior art; this means that within the same time, the present invention can complete more image stitching tasks with lower power consumption, demonstrating higher energy efficiency.

[0154] In summary, through a series of innovative steps and technical means, the present invention not only significantly reduces power consumption, improves the accuracy of feature point detection and matching, but also greatly shortens the stitching time and enhances the quality of the stitched image; especially in terms of the energy efficiency ratio, the present invention shows excellent performance, proving its creativity and novelty in practical applications; these improvements not only solve the deficiencies existing in the prior art, but also provide new ideas and directions for the development of future image stitching technologies.

[0155] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered within the scope of the claims of the present invention.

Claims

1. An FPGA-based image stitching method, characterized in that: Including: Obtain multiple images to be stitched from an image sensor, and analyze and record the image features of each image; Specifically: Use an RGB-D camera to collect multiple images to be stitched; Calculate the complexity of the image according to the image features, evaluate the power consumption required to process the image according to the complexity of the image, and determine the initial processing strategy accordingly; Dynamically adjust the processing strategy according to the power consumption and image complexity; Specifically: For simple images, when it is detected that the power consumption is lower than the set threshold and the FPGA resource utilization rate does not reach the upper limit, switch from the original fast feature detection algorithm to the SURF algorithm; increase the number of matching points by applying multi-scale detection; use the RANSAC algorithm combined with Levenberg-Marquardt to estimate the homography matrix to obtain the stitching result; For medium-complexity images, keep the initially determined processing strategy unchanged, and at the same time, continuously track the power consumption change; once the power consumption exceeds the preset limit, trigger an alarm and automatically reduce the number of feature points or simplify the matching process; dynamically adjust the resolution of the input image according to actual needs; For complex images, when the power consumption exceeds the budget, simplify the original SURF algorithm to ORB; only perform feature point detection at the edges and corners of the image, and ignore flat areas; divide the large-size image into small sub-blocks, and process them in parallel on different logic units of the FPGA, and then summarize the results to complete the final stitching task; adopt a multi-layer fusion technology to generate a preliminary stitching result, and then refine the boundaries and overlapping areas; Extract key feature points from each image to generate feature descriptors; Specifically: For each detected key point, divide the area around each key point into multiple sub-regions. In each sub-region, calculate the distribution of the local gradient direction to form a gradient direction histogram; calculate the depth difference between adjacent sub-regions to form a depth difference histogram; connect the gradient direction histograms and depth difference histograms of all sub-regions to form a feature descriptor vector; Match the generated feature descriptors, record the matching point pairs, and calculate the transformation parameters between the images according to the matching results; Perform geometric transformation and fusion on each image for the calculated transformation parameters to generate the final stitched image.

2. The FPGA-based image stitching method according to claim 1, wherein: The step of obtaining multiple images to be stitched from an image sensor, and analyzing and recording the image features of each image is specifically further: Calculate the average brightness and contrast of each image. The formula for calculating the average brightness of each image is: where M represents the total number of pixels, and Ω k represents the set of all pixel points of the k-th image, and R(x, y), G(x, y), and B(x, y) respectively represent the component values of red, green, and blue at the pixel point (x, y).

3. The method for image stitching based on FPGA according to claim 2, wherein: The step of calculating the complexity of the image according to the image features, evaluating the power consumption required to process the image according to the complexity of the image, and determining the initial processing strategy accordingly is specifically: Calculate the complexity D of an image based on the average brightness and contrast of each image k , and the expression is: D k = α·C k + β·(σ L / L k ) Among them, D k represents the complexity of the k-th image, α represents the weight coefficient of contrast, β represents the weight coefficient of the change rate, and σ L represents the standard deviation of the brightness of the k-th image; C k represents the contrast of the k-th image; According to the complexity D of the image k The images are classified into three categories: simple images, moderately complex images, and complex images; Set complexity thresholds T1 and T2; When D k <T1, it is a simple image and a low-precision calculation strategy is adopted; When T1 ≤ D k < T2, it is a medium-complexity image, and a medium-precision calculation strategy is adopted; When D k ≥ T2, it is a complex image and is suitable for adopting a high-precision calculation strategy; According to the complexity D of the image k Calculate the power consumption P required to process the image k , and the expression is: Among them, P k represents the power consumption required to process the k-th image, and P base represents the base power consumption during operation, and η represents the weight coefficient of complexity.

4. The FPGA-based image stitching method according to claim 3, characterized in that: The step of extracting key feature points from each image to generate feature descriptors is specifically further: Use the SURF algorithm to detect key points in the image; Normalize the feature descriptor vector to obtain the normalized feature descriptor intensity F(r), and the expression is: Among them, F(r) represents the feature descriptor strength of the r-th key point after normalization, w i represents the weight factor of the i-th sub-region, h i represents the histogram of gradient directions value of the i-th sub-region, b i represents the histogram of depth difference values of the i-th sub-region, and n represents the total number of divided sub-regions.

5. The FPGA-based image stitching method according to claim 4, wherein: The step of matching the generated feature descriptors, recording the matching point pairs, and calculating the transformation parameters between the images according to the matching results is specifically: Feature point matching is performed using the bidirectional nearest neighbor algorithm based on the extracted key feature points and their descriptors; A scoring function is constructed based on the key feature points and their descriptors to evaluate the reliability of each pair of matching points. The expression is: Among them, S(p δ , q j ) represents the reliability score value of the matching point pair p δ and q j . p δ represents the δ-th key point extracted from the first image, q j represents the j-th key point extracted from the second image, d(p δ , q j ) represents the Euclidean distance between the two descriptors of p δ and q j . λ represents the parameter controlling the slope of the sigmoid function, μ represents the average distance, ΔR(p δ , q j ) represents the depth difference between p δ and q j , γ represents the weight factor, and σ R represents the standard deviation of the depth differences of all matching point pairs; For the reliable matching point pairs screened by the scoring function, the RANSAC algorithm is used to estimate the geometric transformation parameters between the images, and the homography matrix H is obtained.

6. The FPGA-based image stitching method according to claim 5, characterized in that: The geometric transformation of each image is performed on the calculated transformation parameters and fused to generate the final stitched image. Specifically: The FPGA is used to perform affine transformation on each image for stitching to obtain the stitched image; The weighted average fusion method is used to blend the overlapping regions in the stitched image to eliminate the stitching traces and generate the final stitched image.

7. An FPGA-based image stitching system, based on the FPGA-based image stitching method according to any one of claims 1 to 6, characterized in that: Including: An acquisition module, an adjustment module, a descriptor module, a matching module, and a fusion module: The acquisition module is responsible for obtaining multiple images to be stitched from the image sensor, analyzing and recording the color information RGB and depth information Depth of each image, and calculating the average brightness and contrast of the images; The adjustment module is used to calculate the image complexity according to the image features, evaluate the power consumption required to process these images according to the complexity, and dynamically adjust the processing strategy; The descriptor module uses the SURF algorithm to detect the key feature points in the image and generate corresponding feature descriptors; The matching module uses the bidirectional nearest neighbor algorithm for feature point matching and estimates the geometric transformation parameters between the images based on the matching results using the RANSAC algorithm; The fusion module is responsible for using the FPGA to accelerate the execution of affine transformation or other geometric transformation operations to transform all images into the same coordinate system, perform weighted average fusion on the overlapping regions, eliminate the stitching traces, and generate the final stitched image.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the FPGA-based image stitching method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the FPGA-based image stitching method according to any one of claims 1 to 6 are implemented.

Images