Image splicing method and system based on FPGA (Field Programmable Gate Array), medium and equipment

Abstract

The invention discloses an image splicing method and system based on an FPGA, a medium and equipment, and relates to the field of computer vision, and the method comprises the steps: obtaining a plurality of to-be-spliced images from an image sensor, and analyzing and recording the image features of each image; calculating the complexity of the image according to the image features, evaluating the power consumption required for processing the image according to the complexity of the image, and determining an initial processing strategy according to the power consumption; dynamically adjusting a processing strategy according to the power consumption and the image complexity; key feature points are extracted from each image, and feature descriptors are generated; matching the generated feature descriptors, recording matching point pairs, and calculating transformation parameters between the images according to a matching result; and performing geometric transformation and fusion of each image on the transformation parameters obtained by calculation to generate a final spliced image. According to the method, the RGB-D camera is used for acquiring the color and depth information of the image, so that the accuracy of geometric transformation is enhanced, and richer data support is provided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image stitching method, system, medium and device based on FPGA. Background Art

[0002] Image stitching technology, as an important part of computer vision, has developed rapidly in recent years with the advancement of hardware computing power and algorithms. Traditional methods rely on CPU processing to achieve seamless fusion of multiple images through feature point detection, descriptor generation, matching and geometric transformation. However, when facing application scenarios with high real-time requirements for video streams, CPU methods show problems of high computing resource consumption and long processing time when processing large-scale data sets or high-resolution images. Although GPUs have significantly improved processing speeds with their powerful parallel computing capabilities, their high power consumption limits their applications in embedded systems and mobile devices. In addition, the original intention of GPU architecture design was not for image stitching tasks, and there are limitations in flexibility and customization. In contrast, FPGAs, with their highly flexible and efficient characteristics, provide better performance-power ratios and allow logic circuits to be customized according to specific needs. They are very suitable for applications that require a balance between high performance and low power consumption.

[0003] Although the existing image stitching technology can meet basic needs, it still faces challenges when processing complex images. The traditional stitching method adopts a fixed processing strategy and cannot dynamically adjust the calculation accuracy and resource allocation, resulting in resource waste in simple image processing. In addition, the quality of complex image processing may be affected by insufficient resources. In addition, most solutions fail to fully utilize the color and depth information provided by RGB-D cameras, especially the application value in geometric transformation is ignored. The existing algorithms pay more attention to the quality of the final result and pay less attention to the power consumption management of the whole process, which is particularly important for resource-constrained environments. The present invention solves the above problems through an FPGA-based method, combined with dynamic processing strategy adjustment, full utilization of RGB-D information and efficient image fusion technology, and realizes efficient and low-power image stitching, which is particularly suitable for real-time processing of high-resolution images. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an FPGA-based image stitching method to solve the problem that the traditional stitching method adopts a fixed processing strategy and cannot dynamically adjust the calculation accuracy and resource allocation, resulting in resource waste in simple image processing, while the quality of complex image processing may be affected due to insufficient resources.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides an image stitching method based on FPGA, which comprises: Acquire multiple images to be stitched from an image sensor, and analyze and record image features of each image; Calculate the complexity of the image based on the image features, evaluate the power consumption required to process the image based on the image complexity, and decide the initial processing strategy accordingly; Dynamically adjust processing strategies based on power consumption and image complexity; Extract key feature points from each image and generate feature descriptors; Match the generated feature descriptors, record the matching point pairs, and calculate the transformation parameters between the images based on the matching results; The calculated transformation parameters are used to perform geometric transformation on each image and fuse them to generate the final stitched image.

[0007] As a preferred solution of the FPGA-based image stitching method of the present invention, wherein: the method acquires multiple images to be stitched from the image sensor, and analyzes and records the image features of each image, and the specific steps are: Use an RGB-D camera to capture multiple images to be stitched; Calculate the average brightness and contrast of each image. The average brightness of each image is calculated as: ; in, Indicates The average brightness of an image, Represents the total number of pixels, Indicates The set of all pixels of an image, , and Respectively represent the pixels The red, green, and blue component values; The contrast of each image is calculated as: ; in, Indicates The contrast of the image, Indicates Image at pixel point The brightness value, The index variable representing the image.

[0008] As a preferred solution of the FPGA-based image stitching method of the present invention, wherein: the complexity of the image is calculated according to the image features, the power consumption required for processing the image is evaluated according to the complexity of the image, and the initial processing strategy is determined accordingly, the specific steps are: Calculate the complexity of the image based on the average brightness and contrast of each image , the expression is: ; in, Indicates The complexity of an image indicates the richness of image features. represents the weight coefficient of contrast, The weight coefficient representing the rate of change, Indicates The standard deviation of the average brightness of the image; According to the complexity of the image Classify images into three categories: simple images, medium-complexity images, and complex images; Set complex thresholds T1 and T2; when When < T1, it is a simple image, which means that the image has few features, uniform brightness, low contrast, and a low-precision calculation strategy is used; When T1≤ When < T2, it is a medium-complex image, indicating that the image has certain characteristic information, the brightness and texture changes are moderate, and a medium-precision calculation strategy is adopted; when When ≥T2, it is a complex image, which means that the image features are rich, the brightness changes greatly, and the contrast is high, which is suitable for high-precision calculation strategies; According to the complexity of the image Calculate the power required to process an image , the expression is: ; in, Indicates processing The power consumption required for an image, Indicates the basic power consumption of the operation. The weight coefficient representing the complexity.

[0009] As a preferred solution of the FPGA-based image stitching method of the present invention, the specific steps of dynamically adjusting the processing strategy according to power consumption and image complexity are as follows: For simple images, when monitoring power consumption When the value is significantly lower than the set threshold and the FPGA resource utilization rate does not reach the upper limit, the original fast feature detection algorithm is switched to the SURF algorithm. Improve stitching accuracy by applying multi-scale detection to increase the number of matching points; Use RANSAC combined with the local optimization algorithm Levenberg-Marquardt to estimate the homography matrix to obtain higher quality stitching results; For moderately complex images, the initially selected computing strategy is maintained while tracking power consumption changes in real time. Once power consumption exceeds the preset limit, an early warning is triggered and the number of feature points is automatically reduced or the matching process is simplified. Dynamically adjust the resolution of input images according to actual needs; For complex images, when power consumption exceeds the budget, the SURF algorithm originally used to generate complex feature descriptors is simplified to the lighter ORB algorithm to reduce the computational burden. Only perform feature point detection on the edges and corners of the image, ignoring flat areas, reducing the overall computational effort; Divide the large-size image into small sub-blocks, process them in parallel on different logic units of the FPGA, and then summarize the results to complete the final stitching task; Multi-layer fusion technology is used to generate preliminary stitching results, and then refine the boundaries and overlapping areas.

[0010] As a preferred solution of the FPGA-based image stitching method of the present invention, the key feature points are extracted from each image to generate feature descriptors, and the specific steps are as follows: The accelerated robust feature SURF algorithm is used to detect key points in the image; For each detected key point, the area around each key point is divided into multiple sub-regions, and the gradient direction histogram mHoG is calculated in each sub-region and combined with the depth difference histogram DDH; Concatenate the gradient direction histograms and depth difference histograms of all sub-regions to form the final feature descriptor vector; Normalize the feature descriptor vector to obtain the normalized feature descriptor strength , the expression is: ; in, Represents the normalized The feature descriptor strength of key points, Indicates The weight factor of each sub-region, Indicates The gradient direction histogram value of the sub-region, Indicates The depth difference histogram value of the sub-region, Indicates the total number of sub-areas divided.

[0011] As a preferred solution of the FPGA-based image stitching method of the present invention, the generated feature descriptors are matched, matching point pairs are recorded, and the transformation parameters between images are calculated according to the matching results. The specific steps are as follows: Based on the extracted key feature points and their descriptors, the bidirectional nearest neighbor BNN algorithm is used to match the feature points; Based on the key feature points and their descriptors, a scoring function is constructed to evaluate the reliability of each pair of matching points. The expression is: ; in, Represents matching point pairs and The reliability score of Indicates the first image extracted Key points, Indicates the first Key points, express and The Euclidean distance between two descriptors, Represents the parameter that controls the slope of the sigmoid function, represents the average distance, express and The depth difference between represents the weight factor, Represents the standard deviation of the depth differences of all matching point pairs; Subsequently, for the reliable matching point pairs screened by the scoring function, the random sampling consensus algorithm RANSAC is used to estimate the geometric transformation parameters between the images to obtain the homography matrix H.

[0012] As a preferred solution of the FPGA-based image stitching method of the present invention, the specific steps of performing geometric transformation on each image and fusing the calculated transformation parameters to generate the final stitched image are as follows: For each image to be stitched, use the homography matrix H to perform an affine transformation on it, and apply this homography matrix to each image to be stitched so that all images can be in the same coordinate system; Using FPGA to perform affine transformation on each image to stitch them together and obtain a stitched image; The weighted average fusion method is used to blend the overlapping areas in the stitched images, eliminate the stitching traces, and generate the final stitched image.

[0013] In a second aspect, the present invention provides an image stitching system based on FPGA, including: an acquisition module, an adjustment module, a descriptor module, a matching module and a fusion module: The acquisition module is responsible for acquiring multiple images to be stitched from the image sensor, analyzing and recording the color information RGB and depth information Depth of each image, and calculating the average brightness and contrast of the image; The adjustment module is used to calculate the complexity of the image according to the image features, and to evaluate the power consumption required for processing the images according to the complexity, and to dynamically adjust the processing strategy; The descriptor module uses methods such as the SURF algorithm to detect key feature points in the image and generate corresponding feature descriptors; The matching module uses a bidirectional nearest neighbor (BNN) algorithm to match feature points, and uses a RANSAC algorithm to estimate geometric transformation parameters between images based on the matching results; The fusion module is responsible for using FPGA to accelerate the execution of affine transformation or other geometric transformation operations to convert all images into the same coordinate system, perform weighted average fusion on overlapping areas, eliminate stitching traces, and generate a seamless and visually coherent final stitched image.

[0014] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the FPGA-based image stitching method as described in the first aspect of the present invention is implemented.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the FPGA-based image stitching method as described in the first aspect of the present invention is implemented.

[0016] The beneficial effects of the present invention are as follows: the present invention acquires the color and depth information of the image by using an RGB-D camera, thereby enhancing the accuracy of the geometric transformation and providing richer data support; secondly, by calculating the image complexity and evaluating the power consumption, the optimal allocation of resources is achieved, ensuring that simple images do not waste resources and complex images are fully processed, thereby improving the overall efficiency of the system; the dynamic adjustment of the processing strategy further optimizes the balance between real-time performance and energy efficiency, and is particularly suitable for resource-constrained application scenarios; the SURF algorithm is used to extract key feature points, which not only speeds up the detection speed, but also improves the stability and reliability of matching; the bidirectional nearest neighbor BNN algorithm is combined with the scoring function to reduce the mismatching phenomenon and ensure the accuracy of matching; the RANSAC algorithm enhances the robustness of the transformation parameter estimation; finally, the affine transformation is accelerated by using FPGA hardware, and the overlapping area is processed by the weighted average fusion method, so as to achieve seamless and visually coherent high-quality spliced ​​images; in summary, the present invention greatly improves the speed and quality of image splicing while ensuring high precision and low power consumption, and is suitable for occasions of real-time processing of high-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 This is a flow chart of the FPGA-based image stitching method in Example 1.

[0019] Figure 2 Schematic diagram of the FPGA-based image stitching system in Example 1. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides an image stitching method based on FPGA, comprising the following steps: S1, acquiring multiple images to be stitched from an image sensor, and analyzing and recording image features of each image; Use an RGB-D camera (Intel RealSense) to collect multiple images to be stitched. The reason for choosing an RGB-D camera here is that it can simultaneously obtain the color information (RGB) and depth information (Depth) of the image, providing richer data support for subsequent geometric transformations; The image features of each image include the average brightness and contrast of each image; Calculate the average brightness and contrast of each image. The average brightness of each image is calculated as: ; in, Indicates The average brightness of an image, Represents the total number of pixels, Indicates The set of all pixels of an image, , and Respectively represent the pixels The red, green, and blue component values ​​are in the range of [0,255]. , and are the weight coefficients of red, green and blue components respectively; The contrast of each image is calculated as: ; in, Indicates The contrast of an image indicates the difference in brightness and darkness of the image. Indicates Image at pixel point The brightness value, The index variable representing the image, The value range of is from 1 to N, where N represents the number of images collected; The use of an RGB-D camera (Intel RealSense) in step S1 to acquire multiple images to be stitched and analyze and record image features is an innovation. The depth information provided by the RGB-D camera provides richer data support for subsequent geometric transformations, which has obvious advantages over traditional cameras that only provide two-dimensional information. Specifically, depth information can better understand the spatial structure of the scene, thereby improving the accuracy and quality of image stitching. This technology is particularly important in the fields of three-dimensional reconstruction and augmented reality.

[0024] S2, calculating the complexity of the image according to the image features, evaluating the power consumption required for processing the image according to the complexity of the image, and determining the initial processing strategy accordingly; Calculate the complexity of the image based on the average brightness and contrast of each image , the expression is: ; in, Indicates The complexity of an image indicates the richness of image features. represents the weight coefficient of contrast, The weight coefficient representing the rate of change, Indicates Standard deviation of the average brightness of an image According to the complexity of the image The images are classified into three categories: simple images, moderately complex images, and complex images Set complex thresholds T1 and T2 When < T1, it is a simple image, indicating that the image has fewer features, uniform brightness, low contrast, and a low-precision calculation strategy is adopted When T1 ≤ < T2, it is a moderately complex image, indicating that the image has certain feature information, moderate brightness and texture changes, and a medium-precision calculation strategy is adopted When ≥ T2, it is a complex image, indicating that the image has rich features, large brightness changes, high contrast, and is suitable for adopting a high-precision calculation strategy According to the complexity of the image Calculate the power consumption required to process the image ; ; Among them, Indicates the power consumption required to process the th image, Indicates the basic power consumption during operation, Indicates the weight coefficient of complexity

[0025] S3. Dynamically adjust the processing strategy according to the power consumption and image complexity For simple images, when it is detected that the power consumption is significantly lower than the set threshold and the FPGA resource utilization rate has not reached the upper limit, switch from the original fast feature detection algorithm to the SURF algorithm Increase the number of matching points by applying multi-scale detection, thereby improving the accuracy of stitching Use RANSAC combined with the Levenberg-Marquardt of the local optimization algorithm to estimate the homography matrix and obtain a higher-quality stitching result For moderately complex images, keep the initially selected calculation strategy unchanged and simultaneously track the power consumption changes in real time; once the power consumption exceeds the preset limit, trigger an alarm and automatically reduce the number of feature points or simplify the matching process Dynamically adjust the resolution of the input image according to actual needs For complex images, when the power consumption exceeds the budget, simplify the originally used algorithm SURF for generating complex feature descriptors to the lighter-weight ORB to reduce the computational burden Perform feature point detection only at the edges and corners of the image, ignoring flat areas, to reduce the overall computational amount Divide the large-size image into small sub-blocks, process them in parallel on different logic units of the FPGA, and then summarize the results to complete the final stitching task; Multi-layer fusion technology is used to generate preliminary stitching results, and then refine the boundaries and overlapping areas.

[0026] The core of the present invention is to dynamically adjust the processing strategy according to image complexity and power consumption. Traditional methods usually adopt fixed processing strategies, which leads to resource waste or insufficient performance. The present invention introduces an image complexity evaluation mechanism to adaptively adjust the calculation accuracy and resource allocation according to different types of images, thereby optimizing the overall performance. The proposed method solves the problem in the prior art that the processing strategy cannot be flexibly adjusted according to the image characteristics. Specifically, when processing simple images, the accuracy is improved without increasing power consumption. For complex images, when the power consumption exceeds the budget, the overall performance is guaranteed by reducing the calculation accuracy to avoid the deterioration of stitching quality due to insufficient resources.

[0027] S4, extracting key feature points from each image and generating feature descriptors; The accelerated robust feature SURF algorithm is used to detect key points in the image. SURF is a fast and reliable feature point detector that locates stable key points by finding the extreme points of the Hessian matrix in spaces of different scales. SURF is selected because it has good invariance to rotation and scale changes. Compared with the traditional SIFT algorithm, SURF is faster and more suitable for real-time processing requirements. For each detected key point, the area around each key point is divided into multiple sub-regions, and the gradient direction histogram mHoG is calculated in each sub-region and combined with the depth difference histogram DDH. Specifically: In each sub-region, the Sobel operator is applied to calculate the gradient magnitude and direction of each pixel, and the gradient direction is quantized into discrete angle intervals, for example, every 20 degrees between 0 and 360 degrees is used as an interval. In this way, the continuous direction value can be mapped to a finite number of discrete intervals. A histogram is created for each sub-region, in which each bin corresponds to a quantized gradient direction interval. Then, all pixels in the sub-region are traversed, and the corresponding gradient magnitudes are accumulated into the corresponding bins according to their gradient directions. The final gradient direction histogram reflects the gradient distribution in different directions in the sub-region. Get the depth map of each image from the RGB-D sensor, select a pair of adjacent sub-regions, calculate the average depth between them, and then find the depth difference between the two sub-regions. This depth difference is taken as the difference between the two. Set a series of discrete depth difference intervals and count the number of samples falling within these intervals to construct a depth difference histogram DDH. Finally, the HoG and DDH of each sub-region are connected to form a comprehensive feature description. This comprehensive feature vector can simultaneously express local texture information (provided by HoG) and three-dimensional structure information (provided by DDH); Concatenate the gradient direction histograms and depth difference histograms of all sub-regions to form the final feature descriptor vector; Normalize the feature descriptor vector to obtain the normalized feature descriptor strength , the expression is: ; in, Represents the normalized The feature descriptor strength of key points, Indicates The weight factor of each sub-region, Indicates The gradient direction histogram value of the sub-region, Indicates The depth difference histogram value of the sub-region, Indicates the total number of divided sub-areas; By using the SURF algorithm to detect feature points and combining the depth difference histogram to generate feature descriptors, the present invention not only significantly improves the effect and efficiency of image stitching, but also solves the technical limitations of traditional methods. This method is particularly suitable for applications that require high-quality image stitching, such as panoramic photography and virtual reality. Normalized feature descriptor strength Used to measure the saliency of descriptors and provide better stability and accuracy during matching.

[0028] S5, matching the generated feature descriptors, recording the matching point pairs, and calculating the transformation parameters between the images according to the matching results; Based on the extracted key feature points and their descriptors, the bidirectional nearest neighbor BNN algorithm is used for feature point matching. The traditional one-way nearest neighbor method will lead to false matching, while BNN can more accurately screen out the true matching point pairs by checking the nearest neighbor relationship in two directions. BNN is selected because it can reduce the possibility of false matching while maintaining matching accuracy. Based on the key feature points and their descriptors, a scoring function is constructed to evaluate the reliability of each pair of matching points. The expression is: ; in, Represents matching point pairs and The reliability score of Indicates the first image extracted Key points, Indicates the first Key points, express and The Euclidean distance between two descriptors, Represents the parameter that controls the slope of the sigmoid function, represents the average distance, express and The depth difference between represents the weight factor, Represents the standard deviation of the depth differences of all matching point pairs; The value range of is between [0,1]. The larger the value, the higher the matching reliability, which helps to select the best matching point pair from many candidate matches. Subsequently, for the reliable matching point pairs screened by the scoring function, the random sampling consensus algorithm RANSAC is used to estimate the geometric transformation parameters between the images to obtain the homography matrix H. The reason for choosing RANSAC is that it has good resistance to noise and outliers, which can ensure that the final transformation parameters are accurate enough.

[0029] S6, performing geometric transformation on each image and fusing the calculated transformation parameters to generate a final stitched image; For each image to be stitched, use the homography matrix H to perform an affine transformation on it. Apply this homography matrix to each image to be stitched so that all images can be in the same coordinate system. In order to improve efficiency and accuracy, we map these transformation operations to FPGA hardware and use its parallel processing capabilities to accelerate the calculation process; Using FPGA to perform affine transformation on each image to obtain the stitched image. These transformation operations will adjust the position, size and angle of the images according to the homography matrix so that they can be seamlessly stitched together. The weighted average fusion method is used to blend the overlapping areas in the stitched images, eliminate the stitching traces, and generate the final stitched image.

[0030] This embodiment also provides an FPGA-based image stitching system, including: an acquisition module, an adjustment module, a descriptor module, a matching module and a fusion module: The acquisition module is responsible for acquiring multiple images to be stitched from the image sensor, analyzing and recording the color information RGB and depth information Depth of each image, and calculating the average brightness and contrast of the image; An adjustment module, used to calculate image complexity based on image features, and to evaluate the power consumption required to process these images based on the complexity, and to dynamically adjust the processing strategy; The descriptor module uses methods such as the SURF algorithm to detect key feature points in the image and generate corresponding feature descriptors; The matching module uses the bidirectional nearest neighbor (BNN) algorithm to match feature points and uses the RANSAC algorithm to estimate the geometric transformation parameters between images based on the matching results; The fusion module is responsible for using FPGA to accelerate the execution of affine transformation or other geometric transformation operations to convert all images into the same coordinate system, perform weighted average fusion on overlapping areas, eliminate stitching traces, and generate a seamless and visually coherent final stitched image.

[0031] This embodiment also provides a computer device, which is applicable to the case of an FPGA-based image stitching method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the FPGA-based image stitching method proposed in the above embodiment.

[0032] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0033] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the FPGA-based image stitching method proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0034] In summary, the present invention uses an RGB-D camera to obtain the color and depth information of the image, thereby enhancing the accuracy of the geometric transformation and providing richer data support. Secondly, by calculating the complexity of the image and evaluating the power consumption, the optimal allocation of resources is achieved, ensuring that simple images do not waste resources and complex images are fully processed, thereby improving the overall efficiency of the system. The dynamic adjustment of the processing strategy further optimizes the balance between real-time and energy efficiency, and is particularly suitable for resource-constrained application scenarios. The SURF algorithm is used to extract key feature points, which not only speeds up the detection speed, but also improves the stability and reliability of matching. The bidirectional nearest neighbor BNN algorithm is combined with the scoring function to reduce the mismatching phenomenon and ensure the accuracy of the matching; the RANSAC algorithm enhances the robustness of the transformation parameter estimation. Finally, the affine transformation is accelerated by FPGA hardware, and the overlapping area is processed by the weighted average fusion method to achieve seamless and visually coherent high-quality spliced ​​images. In summary, the present invention greatly improves the speed and quality of image splicing while ensuring high precision and low power consumption, and is suitable for real-time processing of high-resolution images.

[0035] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of an image stitching method based on FPGA is provided.

[0036] In this embodiment, a set of multiple high-resolution images captured by an Intel RealSense RGB-D camera is selected as test samples. These images cover different scene complexities, including simple indoor environments, moderately complex natural scenery, and complex building structures, in order to comprehensively evaluate the effectiveness and superiority of the invented method; First, 10 images to be stitched were acquired through an RGB-D camera, and the color information (RGB) and depth information (Depth) of each image were recorded. To ensure the consistency and accuracy of the data, all images were taken under the same conditions, with the same exposure settings and viewing angle changes. Then, the features of each image were analyzed, and the average brightness and contrast were calculated using the formula and These two indicators are quantified separately. This step not only provides the necessary data support for subsequent geometric transformation, but also enables the system to dynamically adjust the processing strategy according to the specific characteristics of the image; Next, the complexity of the image is further calculated based on the calculated average brightness and contrast. By setting thresholds T1 and T2, the images are divided into three categories: simple, medium-complex and complex. For each type of image, the power consumption required for processing is evaluated. , and accordingly decided the initial processing strategy. In this process, special attention was paid to the effective use of resources to ensure the best performance and energy consumption balance in different types of image processing; Subsequently, the SURF algorithm is used to extract key feature points from each image and generate feature descriptors. Compared with the traditional SIFT algorithm, SURF is not only faster, but also has good invariance to rotation and scale changes, which is very suitable for real-time processing needs. The area around each detected key point is divided into multiple sub-areas, and the gradient direction histogram mHoG is calculated in each sub-area, and combined with the depth difference histogram DDH to form the final descriptor vector. This step significantly improves the accuracy and reliability of feature point detection, laying a solid foundation for subsequent matching and transformation steps; Next, the bidirectional nearest neighbor (BNN) algorithm is used to match feature points, and a scoring function is constructed to evaluate the reliability of matching point pairs. The BNN algorithm reduces the possibility of false matching by checking the nearest neighbor relationship in two directions, and the scoring function further ensures the accuracy of the matching. For the reliable matching point pairs that have been screened, the random sampling consensus algorithm (RANSAC) is used to estimate the geometric transformation parameters between images and obtain the homography matrix H. The RANSAC algorithm has good resistance to noise and outliers, ensuring that the final transformation parameters are accurate enough. Finally, by applying the homography matrix H to each image to be stitched for affine transformation, all images are converted to the same coordinate system to achieve seamless stitching. In order to improve efficiency and accuracy, these transformation operations are mapped to the FPGA hardware for execution, and its parallel processing capabilities are used to accelerate the calculation process. In addition, the weighted average fusion method is used to blend the overlapping areas in the stitched images, eliminating the stitching traces and generating the final high-quality stitched image.

[0037] The details are shown in Table 1 below: Table 1 Experimental data record table It can be seen from the data in Table 1 that the present invention has obvious advantages in many aspects: Reduced power consumption: Compared with the prior art, the power consumption of the present invention is reduced by about 22%, that is, from 3.2 watts to 2.5 watts; this is mainly due to the ability to dynamically adjust the processing strategy according to the complexity of the image, ensuring the optimal allocation of resources and avoiding unnecessary energy consumption.

[0038] Increased number of feature points and matching point pairs: The number of feature points extracted by the present invention increased by 25% (from 1,200 to 1,500), and the number of matching point pairs increased by 50% (from 800 to 1,200); this indicates that the application of the SURF algorithm improves the accuracy and reliability of feature point detection and enhances the stability of subsequent matching.

[0039] The stitching time is greatly shortened: The stitching time of the present invention is only half of that of the prior art, shortened from 12 seconds to 6 seconds; this is because the FPGA hardware accelerates the affine transformation operation and optimizes the entire processing flow, thereby significantly improving the processing speed.

[0040] Improved stitching image quality: The stitching image generated by the present invention has higher quality and better visual effect; this is due to the application of the weighted average fusion method, which ensures a smooth transition of the overlapping area and eliminates obvious stitching marks.

[0041] Improved energy efficiency: The energy efficiency of the present invention reaches 0.48 W / frame, which is much higher than 0.27 W / frame in the prior art; this means that in the same time, the present invention can complete more image stitching tasks with lower power consumption, reflecting higher energy efficiency.

[0042] In summary, the present invention, through a series of innovative steps and technical means, not only significantly reduces power consumption and improves the accuracy of feature point detection and matching, but also greatly shortens the stitching time and improves the quality of the stitched image; especially in terms of energy efficiency ratio, the present invention demonstrates excellent performance and proves its creativity and novelty in practical applications; these improvements not only solve the shortcomings of the existing technology, but also provide new ideas and directions for the future development of image stitching technology.

[0043] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An image stitching method based on FPGA, characterized in that: include: Acquire multiple images to be stitched from an image sensor, and analyze and record image features of each image; Calculate the complexity of the image based on the image features, evaluate the power consumption required to process the image based on the image complexity, and decide the initial processing strategy accordingly; Dynamically adjust processing strategies based on power consumption and image complexity; Extract key feature points from each image and generate feature descriptors; Match the generated feature descriptors, record the matching point pairs, and calculate the transformation parameters between the images based on the matching results; The calculated transformation parameters are used to perform geometric transformation on each image and fuse them to generate the final stitched image.

2. The FPGA-based image stitching method according to claim 1, wherein: The specific steps of acquiring multiple images to be stitched from the image sensor and analyzing and recording the image features of each image are as follows: Use an RGB-D camera to collect multiple images to be stitched; Calculate the average brightness and contrast of each image. The average brightness of each image is calculated as: ； in, Indicates The average brightness of an image, Represents the total number of pixels, Indicates The set of all pixels of an image, , and Respectively represent the pixels The red, green, and blue component values; The contrast of each image is calculated as: ； in, Indicates The contrast of the image, Indicates Image at pixel point The brightness value, An index variable representing an image.

3. The FPGA-based image stitching method according to claim 2, characterized in that: The complexity of the image is calculated according to the image features, the power consumption required for processing the image is evaluated according to the complexity of the image, and the initial processing strategy is determined accordingly. The specific steps are as follows: Calculate the complexity of the image based on the average brightness and contrast of each image , the expression is: ； in, Indicates The complexity of an image indicates the richness of image features. represents the weight coefficient of contrast, The weight coefficient representing the rate of change, Indicates The standard deviation of the average brightness of the image; According to the complexity of the image Classify images into three categories: simple images, medium-complexity images, and complex images; Set complex thresholds T1 and T2; When <When T1, it is a simple image, indicating fewer image features, uniform brightness, and low contrast. A low-precision calculation strategy is adopted; When T1 ≤ <T2, it is a medium-complexity image, indicating that the image has certain characteristic information, with moderate brightness and texture changes, and a medium-precision calculation strategy is adopted; when When ≥T2, it is a complex image, which means that the image features are rich, the brightness changes greatly, and the contrast is high, which is suitable for high-precision calculation strategies; According to the complexity of the image Calculate the power required to process an image , the expression is: ； in, Indicates processing The power consumption required for an image, Indicates the basic power consumption of the operation. The weight coefficient representing the complexity.

4. The FPGA-based image stitching method according to claim 3, characterized in that: The specific steps of dynamically adjusting the processing strategy according to power consumption and image complexity are as follows: For simple images, when monitoring power consumption When the value is significantly lower than the set threshold and the FPGA resource utilization rate does not reach the upper limit, the original fast feature detection algorithm is switched to the SURF algorithm. Improve stitching accuracy by applying multi-scale detection to increase the number of matching points; Use RANSAC combined with the local optimization algorithm Levenberg-Marquardt to estimate the homography matrix to obtain higher quality stitching results; For moderately complex images, the initially selected computing strategy is maintained while tracking power consumption changes in real time. Once power consumption exceeds the preset limit, an early warning is triggered and the number of feature points is automatically reduced or the matching process is simplified. Dynamically adjust the resolution of input images according to actual needs; For complex images, when power consumption exceeds the budget, the SURF algorithm originally used to generate complex feature descriptors is simplified to the lighter ORB algorithm to reduce the computational burden. Only perform feature point detection on the edges and corners of the image, ignoring flat areas, reducing the overall computational effort; Divide the large-size image into small sub-blocks, process them in parallel on different logic units of the FPGA, and then summarize the results to complete the final stitching task; Multi-layer fusion technology is used to generate preliminary stitching results, and then refine the boundaries and overlapping areas.

5. The FPGA-based image stitching method according to claim 4, characterized in that: The key feature points are extracted from each image to generate feature descriptors. The specific steps are as follows: The accelerated robust feature SURF algorithm is used to detect key points in the image; For each detected key point, the area around each key point is divided into multiple sub-regions, and the gradient direction histogram mHoG is calculated in each sub-region and combined with the depth difference histogram DDH. Specifically: In each sub-region, the distribution of local gradient directions is calculated to form a gradient direction histogram; Calculate the depth difference between adjacent sub-regions to form a depth difference histogram; Concatenate the gradient direction histograms and depth difference histograms of all sub-regions to form the final feature descriptor vector; Normalize the feature descriptor vector to obtain the normalized feature descriptor strength , the expression is: ； in, Represents the normalized The feature descriptor strength of key points, Indicates The weight factor of each sub-region, Indicates The gradient direction histogram value of the sub-region, Indicates The depth difference histogram value of the sub-region, Indicates the total number of sub-areas divided.

6. The FPGA-based image stitching method according to claim 5, characterized in that: The generated feature descriptors are matched, matching point pairs are recorded, and the transformation parameters between images are calculated according to the matching results. The specific steps are as follows: Based on the extracted key feature points and their descriptors, the bidirectional nearest neighbor BNN algorithm is used to match the feature points; Based on the key feature points and their descriptors, a scoring function is constructed to evaluate the reliability of each pair of matching points. The expression is: ； in, Represents matching point pairs and The reliability score of Indicates the first image extracted Key points, Indicates the first Key points, express and The Euclidean distance between two descriptors, Represents the parameter that controls the slope of the sigmoid function, represents the average distance, express and The depth difference between represents the weight factor, Represents the standard deviation of the depth differences of all matching point pairs; Subsequently, for the reliable matching point pairs screened by the scoring function, the random sampling consensus algorithm RANSAC is used to estimate the geometric transformation parameters between the images to obtain the homography matrix H.

7. The FPGA-based image stitching method according to claim 6, characterized in that: The calculated transformation parameters are used to perform geometric transformation on each image and fuse them to generate a final stitched image. The specific steps are as follows: For each image to be stitched, use the homography matrix H to perform an affine transformation on it, and apply this homography matrix to each image to be stitched so that all images can be in the same coordinate system; Using FPGA to perform affine transformation on each image to stitch them together and obtain a stitched image; The weighted average fusion method is used to blend the overlapping areas in the stitched images, eliminate the stitching traces, and generate the final stitched image.

8. An FPGA-based image stitching system, based on the FPGA-based image stitching method according to any one of claims 1 to 7, characterized in that: include: Acquisition module, adjustment module, descriptor module, matching module and fusion module: The acquisition module is responsible for acquiring multiple images to be stitched from the image sensor, analyzing and recording the color information RGB and depth information Depth of each image, and calculating the average brightness and contrast of the image; The adjustment module is used to calculate the complexity of the image according to the image features, and to evaluate the power consumption required for processing the images according to the complexity, and to dynamically adjust the processing strategy; The descriptor module uses methods such as the SURF algorithm to detect key feature points in the image and generate corresponding feature descriptors; The matching module uses a bidirectional nearest neighbor (BNN) algorithm to match feature points, and uses a RANSAC algorithm to estimate geometric transformation parameters between images based on the matching results; The fusion module is responsible for using FPGA to accelerate the execution of affine transformation or other geometric transformation operations to convert all images into the same coordinate system, perform weighted average fusion on overlapping areas, eliminate stitching traces, and generate a seamless and visually coherent final stitched image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the FPGA-based image stitching method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the FPGA-based image stitching method described in any one of claims 1 to 7 are implemented.

Images