An FPGA-based large-scale image stitching acceleration method, device and medium

Abstract

The application discloses a large-scale image splicing acceleration method and device based on FPGA, and a medium, and the method comprises the following steps: collecting an initial image set to be spliced by a PL hardware acceleration end; performing SIFT feature point extraction on each initial image in the initial image set by a PS software processing end to obtain a key point feature vector; performing feature matching on the key point feature vector by the PS software processing end, completing feature matching on adjacent initial images, and identifying adjacent images; calculating a homography matrix of transformation between different imaging planes according to the feature matching result by the PS software processing end to obtain camera splicing parameters; and performing splicing acceleration processing on each initial image in the initial image set according to the camera splicing parameters by the PL hardware acceleration end to obtain a target image. The embodiment of the application improves the image splicing speed, reduces the calculation power consumption, and can be widely applied to the technical field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a large-scale image stitching acceleration method and device based on FPGA and a medium. BACKGROUND

[0002] With the development of science and technology, human demand for visual perception is constantly improving, and people are pursuing clearer, farther and wider visual experience. The invention of high-definition camera equipment, long-focus lenses and wide-angle lenses enables people to capture visual information with unprecedented clarity and breadth, which greatly enriches the understanding of the world and promotes the development of science and technology. Especially in the AI era, people's requirements for image quality are getting higher and higher. The fields of security monitoring, robot vision and satellite remote sensing all need devices that support long-distance, large field of view and high-resolution images.

[0003] Under the current camera technology level and cost constraints, there is a mutual restriction between the field of view, the detection distance and the resolution.

[0004] In terms of image stitching processing computing platforms, the mainstream solutions can be divided into the following three categories:

[0005] (1) CPU-based computing platform:

[0006] The images captured by multiple cameras are stitched through CPU serial software algorithms. CPU is suitable for wide application due to its high flexibility, strong versatility and short development cycle. However, CPU cannot meet the speed requirements when processing massive high-speed image data, which becomes its main limitation.

[0007] (2) GPU-based computing platform:

[0008] The images captured by multiple cameras are stitched through GPU parallel software algorithms. GPU is good at processing high-data-parallel image data and has been widely used in many applications. Its parallel processing capability is strong and suitable for large-scale image processing tasks. However, the high power consumption of GPU makes it difficult to deploy on edge devices, limiting the application scenarios.

[0009] (3) FPGA-based computing platform:

[0010] Multiple cameras are directly connected to the FPGA platform, and FPGA is responsible for camera control, image acquisition and stitching. FPGA (Field Programmable Gate Array) has the advantages of strong real-time performance, high flexibility and low power consumption, and can combine hardware and software to process multiple image data in parallel. The FPGA solution has small physical volume and low power consumption, and is convenient to deploy in various application scenarios. However, the development threshold of FPGA-based solutions is high, the development cycle is long, and requires higher professional knowledge and technology.

[0011] The current image stitching system based on CPU and GPU general computing platforms has the following main shortcomings:

[0012] (1) Slow stitching speed and low real-time performance: Although CPU and GPU perform well in flexibility and versatility, when processing ultra-high resolution images, the processing speed is relatively slow due to the huge amount of data, making it difficult to achieve real-time stitching. This is a significant limitation for application scenarios that require immediate feedback, such as security monitoring and robot vision.

[0013] (2) High power consumption and large volume: GPU performs well in handling large-scale parallel computing, but its high power consumption and large volume make it unsuitable for deployment in edge devices or portable devices that require low power consumption. Although CPU has relatively low power consumption, its performance is insufficient when processing high-resolution image stitching, making it difficult to meet actual needs.

[0014] Existing image stitching methods and systems based on special chips or FPGAs also have certain limitations:

[0015] Limited applicability: Although FPGA's parallel computing capability and low power consumption characteristics in image processing make it an ideal solution, existing FPGA image stitching systems are mostly used for medium-resolution and field-of-view applications. There is no mature solution for ultra-high resolution, large field of view, and long-distance multi-camera image stitching. SUMMARY

[0016] Therefore, the embodiments of the present application provide a low-power and high-efficiency large-scale image stitching acceleration method, device and medium based on FPGA.

[0017] One aspect of the embodiments of the present application provides a large-scale image stitching acceleration method based on FPGA, comprising the following steps:

[0018] The PL hardware acceleration end collects a set of initial images to be stitched;

[0019] The PS software processing end extracts SIFT feature points from each initial image in the set of initial images to obtain key point feature vectors;

[0020] The PS software processing end performs feature matching on the key point feature vectors to complete feature matching of adjacent initial images;

[0021] The PS software processing end calculates the homography matrix of the transformation between different imaging planes according to the results of the feature matching to obtain camera stitching parameters;

[0022] According to the camera splicing parameters, the PL hardware acceleration end performs splicing acceleration processing on each initial image in the initial image set to obtain a target image.

[0023] Optionally, the PL hardware acceleration end performs splicing acceleration processing on each initial image in the initial image set to obtain a target image according to the camera splicing parameters, including the following steps:

[0024] The parameter transmission module of the PL hardware acceleration end acquires the camera splicing parameters and sends the camera splicing parameters to the mapping module;

[0025] According to the camera splicing parameters, the mapping module calculates source pixel coordinates through splicing pixel coordinates according to the mapping relationship between the splicing image and the source image indicated by the splicing parameters;

[0026] According to the source pixel coordinates, the image data pre-fetching module reads source pixels;

[0027] The source pixels are weighted to obtain splicing pixels, and the splicing pixels are transmitted to the PS software processing end until the transmission of all splicing pixels is completed, and the target image is spliced.

[0028] Optionally, the mapping module calculates source pixel coordinates through splicing pixel coordinates according to the mapping relationship between the splicing image and the source image indicated by the splicing parameters according to the camera splicing parameters, including the following steps:

[0029] According to the camera splicing parameters, the mapping module calculates source pixel coordinates through splicing pixel coordinates according to the mapping relationship between the splicing image and the source image indicated by the splicing parameters;

[0030] According to the splicing pixel coordinates, the pixel mask lookup table existing in the DDR memory of the PS software processing end is read; wherein, the pixel mask lookup table is used to determine the source of the source pixels in the overlapping area between the splicing images during image splicing;

[0031] According to the pixel mask lookup table, the source pixel coordinates are reversely mapped through the splicing pixel coordinates.

[0032] Optionally, the expression of the direction mapping process is:

[0033]

[0034]

[0035] wherein, X, Y, Z are the reference world coordinates; K i , R i are the intrinsic parameters and the rotation matrix relative to the reference world of the i-th camera; s iis the scale of the i th camera; t ix is the scale of the i th camera; t iy is the scale of the i th camera; t iz is the scale of the i th camera; u, v are the stitching image pixel coordinates; x, y are the source image pixel coordinates.

[0036] Optionally, the method further comprises the following steps:

[0037] The generation process of the stitching image pixel coordinates, the process of the direction mapping and the process of weighting the source image pixels to obtain the stitching image pixels are configured in a pipelined mode, so that each process performs parallel operation in a single clock cycle.

[0038] Optionally, the method further comprises the following steps:

[0039] Before sending the camera stitching parameters to the mapping module, the floating-point data format of the camera stitching parameters is fixed-point processed.

[0040] Optionally, the method further comprises the following steps:

[0041] After obtaining the source image pixel coordinates, it is checked whether the image block cached in the image data prefetching module contains the pixels corresponding to the source image pixel coordinates, if yes, the corresponding Bayer data is extracted, and then the RGB pixel data is outputted after interpolation; if not, the matching image block is read from the memory of the PL hardware acceleration end according to the source image pixel coordinates as the source image pixel;

[0042] The image format stored in the memory is changed from the RGB format to the Bayer format, so that each read image block is in the Bayer format; and a Bayer interpolation module is added in the image data prefetching module, so that the final output is the RGB pixel data.

[0043] Another aspect of the embodiment of the application further provides a large-scale image stitching acceleration device based on FPGA, comprising:

[0044] A first module is configured to collect an initial image set to be stitched by a PL hardware acceleration end;

[0045] A second module is configured to extract SIFT feature points from each initial image in the initial image set by a PS software processing end to obtain a key point feature vector;

[0046] A third module is configured to perform feature matching on the key point feature vector by the PS software processing end to complete the feature matching of adjacent initial images.

[0047] A fourth module configured to calculate, by the PS software processing end, a homography matrix of transformation between different imaging planes according to a result of the feature matching, to obtain camera stitching parameters;

[0048] A fifth module configured to perform, by the PL hardware acceleration end, stitching acceleration processing on each of the initial image set according to the camera stitching parameters, to obtain a target image.

[0049] Another aspect of the embodiment of the present application further provides an electronic device including a processor and a memory.

[0050] The memory is configured to store a program.

[0051] The processor executes the program to implement the method as described above.

[0052] Another aspect of the embodiment of the present application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method as described above.

[0053] The embodiment of the present application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method as described above.

[0054] The embodiment of the present application acquires, by a PL hardware acceleration end, an initial image set to be stitched; extracts, by a PS software processing end, a key point feature vector from each of the initial image set; performs, by the PS software processing end, feature matching on the key point feature vector, to complete feature matching on adjacent initial images; calculates, by the PS software processing end, a homography matrix of transformation between different imaging planes according to a result of the feature matching, to obtain camera stitching parameters; and performs, by the PL hardware acceleration end, stitching acceleration processing on each of the initial image set according to the camera stitching parameters, to obtain a target image. The embodiment of the present application improves the image stitching speed and reduces the computing power consumption. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0056] Figure 1An overall step flowchart provided for the embodiment of the present application;

[0057] Figure 2 A schematic diagram of a framework of an image stitching system provided for the embodiment of the present application;

[0058] Figure 3 A schematic diagram of an image stitching parameter generation framework provided for the embodiment of the present application;

[0059] Figure 4 A schematic diagram of an image stitching hardware acceleration framework provided for the embodiment of the present application;

[0060] Figure 5 A schematic diagram of a pipeline framework of a mapping module provided for the embodiment of the present application;

[0061] Figure 6 A schematic diagram of a pixel mask provided for the embodiment of the present application;

[0062] Figure 7 A schematic diagram of a data structure cache structure provided for the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0064] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0065] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting.

[0067] Before the embodiments of the present application are explained in detail, some related technologies involved in the embodiments of the present application are described as follows:

[0068] SI FT feature point extraction: SI FT (Scale-Invariant Feature Transform) is a computer vision algorithm that detects key points in an image and extracts feature vectors that describe these key points.

[0069] K-nearest neighbor (KNN) is a basic classification method that classifies by measuring the distance between different feature values. The idea is that if a sample belongs to a certain class, most of the k most similar (i.e. nearest neighbors) samples in the feature space belong to this class, where K is usually an integer not greater than 20.

[0070] The FPGA-based large-scale image stitching acceleration method provided by the embodiments of the present application relates to the technical field of image processing. The FPGA-based large-scale image stitching acceleration method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application that implements the FPGA-based large-scale image stitching acceleration method, and the like, but is not limited to the above forms.

[0071] The application is operable in numerous general or special computer system environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0072] In view of the problems of the prior art, in order to realize the splicing of ultra-high resolution images and reduce power consumption, the application utilizes the high-efficiency parallel processing capability of the FPGA SoC platform to solve the problems of slow speed, low real-time performance and high power consumption of the prior art in splicing ultra-high resolution images. The hardware parallel processing characteristics of the FPGA significantly improve the speed of image splicing, meet the demand of real-time processing, and complete efficient calculation with low power consumption, so that the system is suitable for application scenarios sensitive to power consumption, such as edge devices and portable devices. Figure 1 The FPGA-based large-scale image splicing acceleration method according to an embodiment of the application includes the following steps:

[0073] An initial image set to be spliced is collected by a PL hardware acceleration end;

[0074] SIFT feature points of each initial image in the initial image set are extracted by a PS software processing end to obtain a key point feature vector;

[0075] Feature matching of the key point feature vector is performed by the PS software processing end to complete feature matching of adjacent initial images;

[0076] A homography matrix of transformation between different imaging planes is calculated according to a result of the feature matching by the PS software processing end to obtain camera splicing parameters;

[0077] The initial images in the initial image set are spliced and accelerated by the PL hardware acceleration end according to the camera splicing parameters to obtain a target image.

[0078] Optionally, the splicing and acceleration of the initial images in the initial image set by the PL hardware acceleration end according to the camera splicing parameters to obtain a target image includes the following steps:

[0079] acquire the camera stitching parameters by the parameter transmission module of the PL hardware acceleration end, and send the camera stitching parameters to a mapping module;

[0080] According to the camera stitching parameters, the mapping module calculates the source image pixel coordinates through the stitching image pixel coordinates according to the mapping relationship between the stitching image and the source image indicated by the stitching parameters;

[0081] According to the source image pixel coordinates, the image data pre-fetching module reads the source image pixels;

[0082] The source image pixels are weighted to obtain stitching image pixels, and the stitching image pixels are transmitted to the PS software processing end until the transmission of all stitching image pixels is completed, and the target image is obtained by stitching.

[0083] Optionally, the mapping module calculates the source image pixel coordinates through the stitching image pixel coordinates according to the mapping relationship between the stitching image and the source image indicated by the stitching parameters, according to the camera stitching parameters, includes the following steps:

[0084] According to the camera stitching parameters, the mapping module calculates the source image pixel coordinates through the stitching image pixel coordinates according to the mapping relationship between the stitching image and the source image indicated by the stitching parameters, includes the following steps:

[0085] According to the stitching image pixel coordinates, the pixel mask lookup table existing in the PS software processing end DDR memory is read; wherein, the pixel mask lookup table is used to determine the source of the source pixels in the overlapping area between the stitching images during image stitching;

[0086] According to the pixel mask lookup table, the source image pixel coordinates are reversely mapped through the stitching image pixel coordinates.

[0087] Optionally, the expression of the direction mapping process is:

[0088]

[0089]

[0090] wherein, X, Y, Z are the reference world coordinates; K i , R i are the intrinsic parameters and the rotation matrix relative to the reference world of the i-th camera respectively; s i is the scale of the i-th camera; t ix , t iy , t iz are the coordinate offsets of the i-th camera; u, v are the stitching image pixel coordinates; x, y are the source image pixel coordinates.

[0091] Optionally, the method further includes the following steps:

[0092] The generation process of the stitched image pixel coordinates, the process of the direction mapping and the process of weighting the source image pixels to obtain the stitched image pixels are configured in a pipelined mode so that each process performs a parallel operation in a single clock cycle.

[0093] Optionally, the method further comprises the following steps:

[0094] Before the camera stitching parameters are sent to the mapping module, the floating point data format of the camera stitching parameters is fixed-pointed.

[0095] Optionally, the method further comprises the following steps:

[0096] After the source image pixel coordinates are obtained, it is checked whether the cached tile in the image data pre-fetching module contains the pixels corresponding to the source image pixel coordinates, if yes, the corresponding Bayer data is extracted, and then RGB pixel data is outputted after interpolation; if not, the matching image block is read from the memory of the PL hardware acceleration end according to the source image pixel coordinates as the source image;

[0097] The image format stored in the memory is changed from the RGB format to the Bayer format so that each read image block is in the Bayer format; and a Bayer interpolation module is added in the image data pre-fetching module so that the final output is the RGB pixel data.

[0098] The specific implementation process of the method of the present application is described in detail below taking a specific application scenario as an example:

[0099] As shown in Figure 2 The image stitching system framework in the FPGA SoC can be divided into an image stitching parameter generation algorithm part implemented in the processing system PS end and an image acquisition and stitching hardware acceleration part implemented in the programmable logic PL end. The device acquires multiple camera images through the PL end, calculates the stitching parameters in the PS end according to the image data, and finally performs image stitching acceleration in the PL end according to the generated stitching parameters to obtain the stitching result.

[0100] The first part of the PS end image stitching parameter generation can be divided into three steps, as shown in Figure 3 The extracted 9 camera images are subjected to SIFT feature point extraction, then subjected to K-NN algorithm (here, the 2-NN algorithm is specifically used in the embodiment of the present application), the adjacent camera images are subjected to feature matching, and finally the camera stitching parameters (i.e. homography matrix of direct transformation of different imaging planes) are obtained.

[0101] The second part of the PL end image stitching hardware acceleration part is as shown in Figure 4The main workflow of this second part is that the PL end sends the splicing parameters to the mapping module through the parameter transmission module, the mapping module calculates the source image pixel coordinates according to the mapping relationship through the splicing image pixel coordinates, then reads the source image pixel according to the source image pixel coordinates through the image data pre-fetching module, finally weights the source image pixel to obtain the splicing image pixel, and finally the result is transmitted to the PS end. When all the splicing graph coordinates are mapped, the splicing process of a graph is completed.

[0102] The initial stage of the overall method of the embodiment of the application has the following detailed process: 9 camera data is decoded into Bayer format image data by the camera data decoding module, is temporarily stored into the PL end DDR through the VDMA, AXI interconnection transmission module 1, then the PS end initiates a read source graph command, reads out the Bayer format image from the PL end DDR, sequentially stores the RGB image into the PS end DDR memory through the VDMA, Bayer interpolation module (to obtain the RGB format image), VDMA, AXI interconnection module 2. Finally, the image splicing parameters are calculated, generated, preprocessed using the source graph, and configured to the mapping module through the parameter transmission module.

[0103] The splicing stage of the embodiment of the application has the following detailed process: 9 camera data is decoded into Bayer format image data by the camera data decoding module, is temporarily stored into the PL end DDR through the VDMA, AXI interconnection transmission module 1. The mapping module generates the splicing graph coordinates according to the parameters, then reads the pixel mask lookup table existing in the PS end DDR memory according to the target graph coordinates (the transmission path is the PS end DDR memory, the AXI interconnection module 2, the AXI4 burst transmission module, the image mask input module), calculates the source graph coordinates according to the pixel mask information, then reads the source graph pixel in the PL end DDR memory according to the source graph coordinates (the transmission path is the PL end DDR memory, the AXI interconnection module 1, the AXI4 burst transmission module, the image data pre-fetching module), finally stores the splicing image pixel obtained by weighting and averaging the pixels into the PS end DDR memory.

[0104] The three functional parts proposed in the application will be described in detail as follows:

[0105] (1) Mapping function module:

[0106] 1) The embodiment of the application adopts image coordinate reverse mapping design: the role of reverse mapping is to calculate the source image pixel coordinates according to the pixel coordinates of the splicing graph. Then the related pixels can be extracted according to the source image pixel coordinates for weighting, and finally the pixel value of the corresponding coordinate of the splicing graph is obtained.

[0107] In the image stitching algorithm framework, after the camera parameters are calculated, image projection transformation is performed. The step involves coordinate transformation. The present application uses reverse mapping design according to the mapping of the stitching image pixel coordinates to the source image pixel coordinates based on the hardware characteristics of FPGA (less on-chip cache, sufficient DDR memory). The actual calculation formula is as follows:

[0108]

[0109]

[0110] Wherein, X, Y, Z are the reference world coordinates; K i , R i are the internal parameters and the rotation matrix of the i-th camera relative to the reference world, respectively; s i is the scale of the i-th camera; t ix , t iy , t iz are the coordinate offsets of the i-th camera; u, v are the stitching image pixel coordinates; x, y are the source image pixel coordinates.

[0111] 2) Full pipeline design:

[0112] Pipeline design according to FPGA characteristics. The image mapping process is a relatively complex process, and pipeline design can divide a complex calculation into multiple parallel simple operations. Finally, a complex calculation is completed in a single clock cycle. The present application, such as Figure 5 The stitching image pixel coordinate generation, source image coordinate mapping and stitching image pixel weighting of the mapping module are all pipeline designed.

[0113] 3) Parameter preprocessing:

[0114] Because FPGA operation on floating-point numbers is relatively time-consuming and resource-consuming than fixed-point numbers. Therefore, the floating-point stitching parameters need to be fixed-pointed.

[0115] (2) Data prefetching function design: including AXI burst transmission module, image mask input module, image data prefetching module.

[0116] 1) Data burst transmission module based on AXI4 protocol:

[0117] The data communication bus used by the hardware platform used by the present application is based on AXI4 protocol. The present application designs a special data burst transmission module based on AXI4 protocol for the module designed by itself, which is used for data transmission between the image stitching hardware acceleration module and the memory.

[0118] 2) Pixel mask input module

[0119] Adjacent images may have overlapping areas when stitched together. This invention innovatively uses a pixel mask lookup table to delineate the source images of these overlapping areas. For example... Figure 6 As shown, there are overlapping areas between the stitched images. When stitching images, it is necessary to determine the source of the pixels in the overlapping area of ​​the stitched image. Here, the present invention can obtain a pixel mask lookup table in advance using the stitching seam algorithm. Each variable in the table represents the source of the pixel at that position (for example, 0 represents that the pixel at that position comes from image 0, and 1 represents that the pixel comes from image 1).

[0120] 3) Image data caching structure based on the spatial locality of image data:

[0121] When implementing reverse mapping using FPGA, a significant issue is the input data bandwidth bottleneck. The source image pixel addresses accessed by the mapping module are always discontinuous, and the random read / write performance of DDR memory is far lower than that of sequential read / write.

[0122] The transformation from the source image to the stitched image is a projection transformation. According to the properties of the transformation matrix, pixels with consecutive coordinates in the target image are also neighbors in the source image when the scales are similar. That is, in this task, the input pixels of the mapping module exhibit a certain degree of spatial locality. Furthermore, generating a pixel in the target image each time requires reading four adjacent integer coordinate pixels from the source image for interpolation. If the read data is discarded after only one use, the input bandwidth will need to be increased several times over for the same stitching speed.

[0123] In summary, if Figure 7 As shown, this invention designs a data structure caching structure. The detailed steps are as follows: After obtaining the pixel coordinates of the source image, check whether the cached image patch within the module contains the pixel corresponding to the coordinates. If it does, extract the corresponding Bayer data, and then output the RGB pixel data after interpolation. If it does not contain the pixel, read the appropriate image patch from the PL-side memory according to the coordinates, and then perform the previous steps.

[0124] This invention reads a small image patch at a time, minimizing repeated readings of the same pixel. Furthermore, because it uses multiple burst transmissions to obtain pixel rows and then stitches them into small image patches, it avoids random readings of single pixels, fully utilizing the burst-continuous transmission performance of DDR memory. To further reduce the input data bandwidth load at the same stitching speed, this invention changes the image format stored in memory from RGB to Bayer format. Each image patch read is in Bayer format, and a Bayer interpolation module is added to the prefetch module, so the final output is still RGB image data. This operation can save up to 2 / 3 of the memory bandwidth.

[0125] (3) Parameter configuration function module design: including parameter generation, parameter preprocessing, parameter transmission module, data path module (reading out the Bayer format image from the PL end DDR, storing the RGB image into the PS end DDR memory via VDMA, Bayer interpolation module, VDMA, AXI interconnection module 2 in sequence).

[0126] Among them, for the key parameter separation: that is, the stitching parameters required by the PL end stitching module, camera resolution, image address information are separated instead of being fixed. Therefore, each time the related parameters are modified, it is not necessary to re-synthesize the project.

[0127] For source image transmission: when the system needs to be reconfigured, the source image can be transmitted to the PS end DDR memory.

[0128] For stitching parameter recalculation: the stitching parameter recalculation capability extended according to the above two functions. That is, the correct stitching parameters can be regenerated when the application scene changes. (Here, the parameters still refer to the K, R, s, t parameters mentioned earlier).

[0129] In some experimental scenarios, the application can stitch 9 camera images with a resolution of 4000*3000, horizontal and vertical field angles of 23° and 17.2° into a fused image with a resolution of 12288*8160, equivalent horizontal and vertical field angles of 62.7° and 43.2° at a speed of 1.82fps. And the calculation power consumption and device volume are less than 6.98 watts and 3 cubic decimeters respectively. Compared with the best prior art, the application has obvious advantages in stitching image resolution, stitching speed, calculation power consumption and device volume. It is easy to deploy in various application scenarios.

[0130] Another aspect of the embodiment of the application also provides a large-scale image stitching acceleration device based on FPGA, comprising:

[0131] The first module is used for collecting an initial image set to be stitched by a PL hardware acceleration end;

[0132] The second module is used for performing SIFT feature point extraction on each initial image in the initial image set by a PS software processing end to obtain a key point feature vector;

[0133] The third module is used for performing feature matching on the key point feature vector by the PS software processing end to complete feature matching of adjacent initial images;

[0134] The fourth module is used for calculating a homography matrix of transformation between different imaging planes according to the feature matching result by the PS software processing end to obtain camera stitching parameters;

[0135] A fifth module configured to perform stitching acceleration processing on each of the initial images in the initial image set according to the camera stitching parameters by the PL hardware acceleration end to obtain a target image.

[0136] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0137] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor realizes the above large-scale image stitching acceleration method based on FPGA when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0138] It can be understood that the contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0139] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program realizes the above large-scale image stitching acceleration method based on FPGA when executed by a processor.

[0140] It can be understood that the contents in the above method embodiments are all applicable to the storage medium embodiments, the storage medium embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0141] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0142] In some alternative embodiments, the function / operations mentioned in the block diagrams can not occur in the order mentioned in the operational illustrations. For example, depending on the involved function / operation, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in reverse order, depending upon the functionality / operations involved. Furthermore, embodiments presented and described in the flowcharts are only examples of implementing the present application. Alternative embodiments are possible where functions described as portions of independent operations are implemented as a combined operation or functions described as a combined operation are implemented as separate operations. The flowcharts can also represent code segments when executed that provide the processes described and / or illustrated therein. Alternatively, the flowcharts can also represent program steps in a procedural, object-oriented, or other programming language object- oriented programming language.

[0143] Furthermore, although the present application is described in the context of functional modules, it is understood that one or more of the functions and / or features described can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the actual implementation is within the routine skill of engineers familiar with the attributes, functions, and internal relationships of the various functional modules disclosed herein. Accordingly, the present application is not limited to the implementations illustrated herein but is intended to be limited only by the claims attached hereto and by their equivalents. It is understood that the specific order or hierarchy of functions and / or features described in the specification is an example and can be performed in any other order or hierarchy.

[0144] The functions described can be implemented in software, firmware, hardware, or any combination thereof. If implemented in software and / or firmware, the functions can be stored or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, or twisted pair, then the coaxial cable, fiber optic cable, or twisted pair are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-Ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0145] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be embodied in non-transitory computer-readable media, executed by one or more computing devices, and / or in any other way. The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer. In this context, a "computer-readable medium" can be any means that can store the program for use by or in connection with the instruction execution system, apparatus, or device.

[0146] The foregoing description of various embodiments of the application has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed, and various modifications and variations are possible in light of the above teachings or can be acquired from practice of the application. For example, while a particular feature of the application can have been described with respect to only one or more embodiments thereof, the feature is not necessarily limited to that one or more embodiments. Rather, applicants have provided various embodiments of the application and combinations thereof and candidates can combine them in various combinations to produce yet other embodiments of the application. It is intended that the specification and examples be considered as exemplary only, with a true scope of the application being indicated by the following claims.

[0147] It is understood that various portions of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiment, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0148] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner.

[0149] While the embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary and are not to be construed as limiting the scope of the application. The scope of the application is defined by the appended claims and their equivalents.

[0150] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the described embodiment, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for accelerating large-scale image stitching based on FPGA, characterized in that, Includes the following steps: The initial set of images to be stitched is acquired by the PL hardware acceleration end; the image stitching system framework on the FPGA SoC can be divided into the image stitching parameter generation algorithm part implemented on the PS software processing end and the image acquisition and stitching hardware acceleration part implemented on the PL hardware acceleration end. The PS software processing unit performs SIFT feature point extraction on each initial image in the initial image set to obtain key point feature vectors; The PS software processing unit performs feature matching on the key point feature vectors to complete feature matching of adjacent initial images; The PS software processing unit calculates the homography matrix of the transformation between different imaging planes based on the feature matching results, and obtains the camera stitching parameters. The PL hardware acceleration terminal performs stitching acceleration processing on each of the initial images in the initial image set according to the camera stitching parameters to obtain the target image; The step of using the PL hardware acceleration terminal to perform accelerated stitching processing on each initial image in the initial image set according to the camera stitching parameters to obtain the target image includes the following steps: The camera stitching parameters are obtained by the parameter transmission module of the PL hardware acceleration terminal, and then sent to the mapping module. Based on the camera stitching parameters, the mapping module calculates the source image pixel coordinates using the pixel coordinates of the stitched image, according to the mapping relationship between the stitched image and the source image indicated by the stitching parameters. Based on the source image pixel coordinates, the source image pixels are read through the image data prefetching module; The source image pixels are weighted to obtain the stitched image pixels, and the stitched image pixels are transmitted to the PS software processing terminal until the transmission of all stitched image pixels is completed, and the target image is obtained by stitching. The step of calculating the pixel coordinates of the source image by the mapping module based on the pixel coordinates of the stitched image and the mapping relationship between the stitched image and the source image indicated by the stitching parameters, according to the camera stitching parameters, includes the following steps: The pixel coordinates of the stitched image are generated based on the camera stitching parameters; The pixel mask lookup table stored in the DDR memory of the PS software processing terminal is read according to the pixel coordinates of the stitched image; wherein, the pixel mask lookup table is used to determine the source pixel source of the overlapping area between the stitched images during image stitching; According to the pixel mask lookup table, the pixel coordinates of the stitched image are reverse-mapped to the pixel coordinates of the source image.

2. The method for accelerating large-scale image stitching based on FPGA according to claim 1, characterized in that, The expression for the reverse mapping process is: , ， in, 、 、 These are the values ​​in the X-axis direction, Y-axis direction, and Z-axis direction of the reference world coordinates, respectively. The first The intrinsic parameters of each camera and the rotation matrix relative to the reference world; For the first Each camera scale; The first The relative coordinate offset of each camera on the X-axis, the relative coordinate offset on the Y-axis, and the relative coordinate offset on the Z-axis; 、 These are the pixel coordinates of the stitched image; 、 These are the pixel coordinates of the source image.

3. The method for accelerating large-scale image stitching based on FPGA according to claim 2, characterized in that, The method further includes the following steps: The process of generating the pixel coordinates of the stitched image, the process of reverse mapping, and the process of weighting the source image pixels to obtain the stitched image pixels are configured in a pipelined mode so that each process can perform parallel operations in a single clock cycle.

4. The method for accelerating large-scale image stitching based on FPGA according to claim 3, characterized in that, The method further includes the following steps: Before sending the camera stitching parameters to the mapping module, the floating-point data format of the camera stitching parameters is converted to fixed-point format.

5. The method for accelerating large-scale image stitching based on FPGA according to claim 4, characterized in that, The method further includes the following steps: After obtaining the source image pixel coordinates, check whether the image patch cached in the image data prefetching module contains the pixel corresponding to the source image pixel coordinates. If yes, extract the corresponding Bayer data, and then output the RGB pixel data after interpolation. If no, read the matching image patch from the memory of the PL hardware acceleration terminal according to the source image pixel coordinates as the source image pixel. The image format stored in memory is changed from RGB to Bayer format so that each image block read is in Bayer format; and a Bayer interpolation module is added to the image data prefetching module so that the final output is RGB pixel data.

6. An apparatus for implementing the FPGA-based large-scale image stitching acceleration method as described in any one of claims 1-5, characterized in that, include: The first module is used to acquire the initial set of images to be stitched by the PL hardware acceleration terminal; The second module is used by the PS software processing end to perform SIFT feature point extraction on each initial image in the initial image set to obtain key point feature vectors. The third module is used by the PS software processing end to perform feature matching on the key point feature vector, and complete the feature matching of adjacent initial images. The fourth module is used by the PS software processing end to calculate the homography matrix of the transformation between different imaging planes based on the feature matching results, and obtain the camera stitching parameters. The fifth module is used by the PL hardware acceleration terminal to perform stitching acceleration processing on each of the initial images in the initial image set according to the camera stitching parameters, so as to obtain the target image.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images