A method for image registration
By using bicubic interpolation on FPGA and feature extraction on CPU, the problem of image registration implemented on FPGA in the prior art is solved, efficient image registration and resource optimization are achieved, and suitable for large-resolution real-time video images deal with.
Patent Information
- Application Number
- CN202111219273.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-10-20
AI Technical Summary
The existing image registration method is complex in computing on the CPU, difficult to implement on the FPGA, and is not suitable for large-resolution real-time video image processing.
The coordinate transformation table of the target image is obtained through FPGA, and the image registration is adopted using the double-cubic interpolation method that is horizontal and then vertical. The CPU is used for feature extraction and feature matching. After generating the coordinate transformation table, it is downsampled to reduce the amount of data and storage resources. Occupancy, realize image registration on FPGA.
It realizes efficient image registration on FPGA, reduces the computational amount and resource usage, meets the real-time registration requirements, and avoids the implementation problems of complex algorithms on FPGAs.
Smart Images

Figure CN114037608B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image registration method. Background Art
[0002] Image registration is the process of matching and superimposing two or more images acquired at different times, with different imaging devices or under different conditions. Among them, different conditions include different climates, illumination, camera positions and angles, etc. Existing image registration methods are divided into two categories, including region-based registration methods and feature-based registration methods. Among them, the region-based registration method mainly uses the grayscale information of the region and does not rely on image features. The feature-based registration method extracts fixed features from the image and completes the registration by matching between features. However, many of these image registration methods have complex calculation processes and are only suitable for running on the central processing unit (CPU). However, the CPU is not capable of processing large-scale data and is not suitable for large-resolution real-time video image display.
[0003] Field Programmable Gate Array (FPGA) has strong large-scale data processing capabilities and fast computing speed, so it has great advantages in image processing. FPGA is often used in binocular cameras to implement image processing. Image registration, as an intermediate link in image processing, also needs to be implemented on FPGA to cooperate with other image processing algorithms. However, the existing registration algorithms are generally more complex and difficult to implement completely on FPGA. This requires an image registration method suitable for implementation on FPGA. Summary of the invention
[0004] The embodiment of the present application provides an image registration method, which is used to implement an image registration method that can meet the requirements of registration effect and processing speed and is suitable for implementation on FPGA.
[0005] In a first aspect, an embodiment of the present application provides an image registration method, comprising:
[0006] A first coordinate transformation table of the target image is obtained through a field programmable gate array FPGA; the first coordinate transformation table includes N elements, where N is a positive integer; wherein one element is the coordinate position of a pixel point in the target image in the image to be registered; the target image and the image to be registered are images of the same object, and the generation conditions of the target image and the image to be registered are different;
[0007] Acquire four pixel points in the image to be registered according to the nth element through the FPGA; the nth element is any element in the first coordinate transformation table, and n is any integer between 1 and N in turn; the four pixel points in the image to be registered are pixel points in the same row of the image to be registered; the row where the four pixel points are located is determined according to the ordinate of the nth element;
[0008] Calculating, by the FPGA, a first pixel value of a pixel point corresponding to an nth element in the image to be registered according to the four pixel points, where n is any integer between 1 and N in sequence;
[0009] Obtaining four first pixel values corresponding to the nth element from the N first pixel values by the FPGA; the column numbers of the four first pixel values are the same as the column number of the nth element, the row numbers of the four first pixel values are adjacent, and the row number of the first first pixel value among the four first pixel values is the same as the row number of the nth element;
[0010] The second pixel value is calculated by the FPGA according to the four first pixel values and the ordinate of the nth element.
[0011] Based on the above scheme, after obtaining the coordinate transformation table of the target image and the image to be registered, bicubic interpolation is performed horizontally and then vertically to achieve the registration of the two images, which can reduce the amount of calculation. Therefore, the image registration can be achieved through FPGA, and the method is simple and easy to implement.
[0012] In a possible implementation, the step of acquiring the first coordinate transformation table of the target image through the FPGA includes:
[0013] Extracting features from the image to be registered and the target image respectively by using a processor CPU to obtain a first feature point set of the image to be registered and a second feature point set of the target image;
[0014] Determining a feature point pair set by the CPU according to a similarity measure; the feature point pair set includes a plurality of feature point pairs, one feature point pair includes a first feature point in the first feature point set and a second feature point in the second feature point set; a similarity measure between the first feature point and the second feature point is greater than or equal to a first threshold;
[0015] The first coordinate transformation table is determined by the CPU according to the set of feature point pairs.
[0016] Based on the above scheme, in the real-time registration of images, the real-time requirements for feature extraction, feature matching and other steps are not as high as those for coordinate transformation and interpolation, but they need to calculate complex functions and iterative operations. Therefore, the CPU can be used to implement feature extraction and feature matching operations, avoiding the problem that FPGA alone cannot meet the needs of complex registration algorithms.
[0017] In a possible implementation manner, determining the first coordinate transformation table according to the feature point pair set includes:
[0018] Determining, by the CPU, a second coordinate transformation table according to the set of feature point pairs;
[0019] The CPU performs k-fold downsampling on the second coordinate transformation table to obtain a third coordinate transformation table; the number of elements in the second coordinate transformation table is k times the number of elements in the third coordinate transformation table; k is an integer greater than 1;
[0020] The third coordinate transformation table is upsampled by k times through the FPGA to obtain the first coordinate transformation table; the number of elements in the first coordinate transformation table is the same as the number of elements in the second coordinate transformation table.
[0021] Based on the above scheme, when the CPU sends the coordinate transformation table to the FPGA, the coordinate transformation table is first downsampled in the CPU, which can reduce the amount of data to be sent. The downsampled table is then sent to the FPGA. After obtaining the table, the FPGA first upsamples it to restore it to its original size before using it. This can reduce the sending time and the storage resource usage of the FPGA, while also meeting the actual alignment requirements.
[0022] In a possible implementation, acquiring four pixel points in the image to be registered according to the horizontal coordinate of the nth element by the FPGA includes:
[0023] The FPGA is used to obtain four pixel points from four RAMs according to the horizontal coordinate of the nth element; wherein one pixel point is obtained from each RAM; the four RAMs respectively store partial pixel points of the image to be registered; and the number of pixel points stored in the four RAMs is the same.
[0024] In one possible implementation, the 4mth column of pixels in the image to be registered are stored in the first RAM, the 4m+1th column of pixels are stored in the second RAM, the 4m+2th column of pixels are stored in the third RAM, and the 4m+3th column of pixels are stored in the fourth RAM, where m is an integer greater than or equal to 0.
[0025] Based on the above solution, since the RAM can only fetch one data at a time, by storing the pixels in four RAMs by columns, the requirement of fetching four data at a time in the bicubic interpolation algorithm can be met to complete horizontal interpolation.
[0026] In a possible implementation, after calculating the first pixel value corresponding to the horizontal coordinate of the nth element according to the four pixel points by the FPGA, the method further includes:
[0027] Cache the obtained N first pixel values into a fifth RAM;
[0028] The acquiring, by the FPGA according to the ordinate of the nth element, four first pixel values among the N first pixel values comprises:
[0029] The FPGA is used to obtain four first pixel values among the N first pixel values from the fifth RAM according to the vertical coordinate of the nth element.
[0030] Based on the above solution, storing the interpolation result in the horizontal direction in the fifth RAM can realize vertical interpolation on the result of horizontal interpolation, which can reduce the amount of FPGA calculation and reduce the occupation of FPGA resources.
[0031] In a possible implementation, after calculating the second pixel value corresponding to the ordinate of the nth element according to the four first pixel values by the FPGA, the method further includes:
[0032] Performing a cross-clock domain operation on the N second pixel values through a first-in first-out memory FIFO; wherein the clock frequency of the N second pixel values after the cross-clock domain operation is the same as the clock frequency of a double data rate synchronous dynamic random access memory DDR;
[0033] The N second pixel values after the cross-clock domain operation are stored in DDR.
[0034] Based on the above solution, the image registration result in RAM can be stored in DDR by first storing the interpolation result in FIFO for cross-clock domain operation.
[0035] In a second aspect, an embodiment of the present application provides an image registration device, including a storage unit and a processing unit:
[0036] The storage unit is used to store computer programs or instructions;
[0037] The processing unit is used to perform the following operations:
[0038] A first coordinate transformation table of the target image is obtained through a field programmable gate array FPGA; the first coordinate transformation table includes N elements, where N is a positive integer; wherein one element is the coordinate position of a pixel point in the target image in the image to be registered; the target image and the image to be registered are images of the same object, and the generation conditions of the target image and the image to be registered are different;
[0039] Acquire four pixel points in the image to be registered according to the nth element through the FPGA; the nth element is any element in the first coordinate transformation table, and n is any integer between 1 and N in turn; the four pixel points in the image to be registered are pixel points in the same row of the image to be registered; the row where the four pixel points are located is determined according to the ordinate of the nth element;
[0040] Calculating, by the FPGA, a first pixel value of a pixel point corresponding to an nth element in the image to be registered according to the four pixel points, where n is any integer between 1 and N in sequence;
[0041] Obtaining four first pixel values corresponding to the nth element from the N first pixel values by the FPGA; the column numbers of the four first pixel values are the same as the column number of the nth element, the row numbers of the four first pixel values are adjacent, and the row number of the first first pixel value among the four first pixel values is the same as the row number of the nth element;
[0042] The second pixel value is calculated by the FPGA according to the four first pixel values and the ordinate of the nth element.
[0043] In a possible implementation, the step of acquiring the first coordinate transformation table of the target image through the FPGA includes:
[0044] Extracting features from the image to be registered and the target image respectively by using a processor CPU to obtain a first feature point set of the image to be registered and a second feature point set of the target image;
[0045] Determining a feature point pair set by the CPU according to a similarity measure; the feature point pair set includes a plurality of feature point pairs, one feature point pair includes a first feature point in the first feature point set and a second feature point in the second feature point set; a similarity measure between the first feature point and the second feature point is greater than or equal to a first threshold;
[0046] The first coordinate transformation table is determined by the CPU according to the set of feature point pairs.
[0047] In a possible implementation manner, determining the first coordinate transformation table according to the feature point pair set includes:
[0048] Determining, by the CPU, a second coordinate transformation table according to the set of feature point pairs;
[0049] The CPU performs k-fold downsampling on the second coordinate transformation table to obtain a third coordinate transformation table; the number of elements in the second coordinate transformation table is k times the number of elements in the third coordinate transformation table; k is an integer greater than 1;
[0050] The third coordinate transformation table is upsampled by k times through the FPGA to obtain the first coordinate transformation table; the number of elements in the first coordinate transformation table is the same as the number of elements in the second coordinate transformation table.
[0051] In a possible implementation, acquiring four pixel points in the image to be registered according to the horizontal coordinate of the nth element by the FPGA includes:
[0052] The FPGA is used to obtain four pixel points from four RAMs according to the horizontal coordinate of the nth element; wherein one pixel point is obtained from each RAM; the four RAMs respectively store partial pixel points of the image to be registered; and the number of pixel points stored in the four RAMs is the same.
[0053] In one possible implementation, the 4mth column of pixels in the image to be registered are stored in the first RAM, the 4m+1th column of pixels are stored in the second RAM, the 4m+2th column of pixels are stored in the third RAM, and the 4m+3th column of pixels are stored in the fourth RAM, where m is an integer greater than or equal to 0.
[0054] In a possible implementation, after calculating the first pixel value corresponding to the horizontal coordinate of the nth element according to the four pixel points by the FPGA, the method further includes:
[0055] Cache the obtained N first pixel values into a fifth RAM;
[0056] The acquiring, by the FPGA according to the ordinate of the nth element, four first pixel values among the N first pixel values comprises:
[0057] The FPGA is used to obtain four first pixel values among the N first pixel values from the fifth RAM according to the vertical coordinate of the nth element.
[0058] In a possible implementation, after calculating the second pixel value corresponding to the ordinate of the nth element according to the four first pixel values by the FPGA, the method further includes:
[0059] Performing a cross-clock domain operation on the N second pixel values through a first-in first-out memory FIFO; wherein the clock frequency of the N second pixel values after the cross-clock domain operation is the same as the clock frequency of a double data rate synchronous dynamic random access memory DDR;
[0060] The N second pixel values after the cross-clock domain operation are stored in DDR.
[0061] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory;
[0062] The memory is used to store computer programs or instructions;
[0063] The processor is used to execute the computer program or instructions in the memory so that the method described in any one of the above aspects is executed.
[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the method described in any one of the above aspects.
[0065] In addition, the beneficial effects of the second to fourth aspects can refer to the beneficial effects described in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application.
[0067] Figure 1 A schematic diagram of the structure of a camera device provided in an embodiment of the present application;
[0068] Figure 2 One of the exemplary flow charts of an image registration method provided in an embodiment of the present application;
[0069] Figure 3 A schematic diagram of downsampling a coordinate transformation table provided in an embodiment of the present application;
[0070] Figure 4 A schematic diagram of bicubic interpolation provided in an embodiment of the present application;
[0071] Figure 5A schematic diagram comparing the simplified function and bicubic function curves provided in the embodiment of the present application;
[0072] Figure 6 A structural diagram of performing row caching of horizontal interpolation results provided in an embodiment of the present application;
[0073] Figure 7 One of the exemplary flow charts of an image registration method provided in an embodiment of the present application;
[0074] Figure 8 A schematic diagram of the structure of an image registration device provided in an embodiment of the present application;
[0075] Fig. 9 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0076] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.
[0077] The terms "first" and "second" in the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.
[0078] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.
[0079] In recent years, multi-source fusion technology has developed rapidly. In the field of monitoring and security, the fusion of infrared and visible light plays an important role. Infrared images are not restricted by weather and light conditions, and can be used in scenes with poor visible conditions and obvious target thermal characteristics. This is a powerful supplement to visible light images. Therefore, binocular cameras based on the fusion of infrared and visible light images play an important role in the field of monitoring and security. However, in binocular cameras that fuse infrared and visible light images, the different installation positions of the two cameras will cause differences in the same picture taken. At this time, it is necessary to align the two images taken by the two cameras by image registration. However, the CPU's ability to process large-scale data is not strong, and it is not suitable for large-resolution real-time video image display. Therefore, how to design an image registration method suitable for implementation on FPGA is urgent.
[0080] In view of this, an embodiment of the present application provides an image registration method. In the method, first, the CPU performs feature extraction, feature matching, and estimation of the transformation model on the target image. After obtaining the coordinate transformation table, the CPU downsamples the coordinate transformation table and sends the downsampled result to the FPGA. Then, the FPGA resamples the target image according to the coordinate transformation table to realize coordinate transformation and interpolation. Thus, the purpose of realizing image registration on the FPGA is achieved.
[0081] The technical solution provided in the embodiment of the present application can be applied to an image registration device, such as a binocular camera based on infrared and visible light image fusion. Figure 1 , is the structure of the image registration device provided in the embodiment of the present application. The device 100 includes a CPU 101, a bus 102 and an FPGA 103. The CPU 101 has a strong ability to schedule, loop and make logical judgments for complex instructions. The data communication between the CPU 101 and the FPGA 103 can be realized through the bus 102. The FPGA 103 has the characteristics of strong ability to process large-scale data and fast computing speed.
[0082] See also Figure 2 , is an exemplary flow chart of an image registration method provided in an embodiment of the present application, which may include the following operations:
[0083] S201, the device obtains a first coordinate transformation table of a target image through FPGA.
[0084] The above-mentioned first coordinate transformation table can be a set of positions of each pixel point in the target image in the image to be registered. Before the device obtains the first coordinate transformation table of the target image through the FPGA, it first calculates the second coordinate transformation table of the target image on the CPU according to certain image feature extraction and feature matching methods. Among them, the second coordinate transformation table includes the position of each pixel point of the target image in the image to be registered. The target image and the image to be registered are images of the same object, and the generation conditions of the target image and the image to be registered are different. Among them, the generation conditions may include shooting time, imaging equipment, shooting angle, shooting position or different weather conditions. In addition, the target image and the image to be registered may also be composite images, which is not limited in this application.
[0085] The steps of calculating the second coordinate transformation table of the target image on the CPU are as follows:
[0086] The device extracts features of the image to be registered and the target image on the CPU respectively, obtains a first feature point set of the image to be registered and a second feature point set of the target image, and then measures the similarity of the feature points in the first feature point set and the second feature point set to determine a feature point pair set. The feature point pair set includes multiple feature point pairs, and a feature point pair includes a first feature point in the first feature point set and a second feature point in the second feature point set. When the similarity measure of the first feature point and the second feature point is greater than or equal to a first threshold, it is determined to be a feature point pair. The device determines the second coordinate transformation table of the target image on the CPU based on the feature point pair set.
[0087] The first threshold may be determined based on an empirical value, which is not specifically limited in this application. Optionally, the similarity measure between the first feature point and the second feature point included in the feature point pair is the largest compared to the similarity measure between the first feature point and other feature points except the second feature point.
[0088] It should be noted that the method for obtaining the coordinate transformation table in the embodiment of the present application is only exemplary, and those skilled in the art may also obtain the coordinate transformation table of the target image and the image to be registered by other means.
[0089] Based on the above scheme, in the real-time registration of images, the real-time requirements for feature extraction, feature matching and other steps are not as high as those for coordinate transformation and interpolation, but they need to calculate complex functions and iterative operations. Therefore, the CPU can be used to implement feature extraction and feature matching operations, avoiding the problem that FPGA alone cannot meet the needs of complex registration algorithms.
[0090] After the device calculates the second coordinate transformation table of the target image on the CPU, it performs k-fold downsampling on it to obtain a third coordinate transformation table, and then sends it to the FPGA, where k is an integer greater than 1.
[0091] After receiving the third coordinate transformation table through FPGA, the device performs k-fold upsampling on it to restore it to the same size as the second coordinate transformation table. The k here can be determined based on empirical values and is not specifically limited in this application.
[0092] See also Figure 3 , which is a schematic diagram of downsampling a coordinate transformation table provided in an embodiment of the present application. Figure 3 In the figure, the 4-fold downsampling of a 32x32 resolution coordinate transformation table is used as an example to illustrate. The shaded part in the figure is the point retained after downsampling, and the non-shaded point is the point lost after downsampling. After the coordinate transformation table with a resolution of 32x32 is downsampled 4 times, the size of the coordinate transformation table becomes 8x8. The upsampling process is the opposite of downsampling. Upsampling is to restore the points lost by downsampling. Optionally, the interpolation calculation of upsampling can use the bilinear interpolation method.
[0093] Based on the above scheme, when the CPU sends the coordinate transformation table to the FPGA, the coordinate transformation table is first downsampled in the CPU, which can reduce the amount of data to be sent. The downsampled table is then sent to the FPGA. After obtaining the table, the FPGA first upsamples it to restore it to its original size before using it. This can reduce the sending time and the storage resource usage of the FPGA, while also meeting the actual alignment requirements.
[0094] S202, the device obtains four pixel points in the image to be registered according to the nth element through FPGA; the nth element is any element in the first coordinate transformation table, and n is any integer between 1 and N in sequence.
[0095] Specifically, when the device obtains four pixel points according to the nth element through FPGA, it can first obtain the pixel points of the row where the integer part of the ordinate of the image to be registered is located according to the integer part of the ordinate of the nth element, and then select the four pixel points in the row with the smallest interpolation value with the abscissa of the nth element as the four pixel points for horizontal interpolation. For example, when the coordinate of the nth element is (2.3, 1.5), first determine to use the first row of pixels according to the ordinate 1.5, and then determine to use the 1st, 2nd, 3rd and 4th columns of the first row according to the abscissa 2.3, and these four pixel points are used as the four pixel points for horizontal interpolation.
[0096] S203, the device calculates the first pixel value of the pixel point corresponding to the nth element in the image to be registered according to the four pixel points through the FPGA.
[0097] In the embodiment of the present application, when calculating the pixel value of the image to be registered through the first coordinate transformation table, since the coordinates in the coordinate transformation table are decimals, that is, the pixel point corresponding to the coordinate may be located between several pixel points in the image to be registered, it is necessary to use interpolation, that is, to use the values of the pixel points around the coordinate to calculate the value of the pixel point corresponding to the coordinate. The embodiment of the present application uses bicubic interpolation for interpolation calculation.
[0098] See also Figure 4 , which is a schematic diagram of bicubic interpolation provided in an embodiment of the present application. Bicubic interpolation is to calculate the pixel value of the point to be interpolated using 16 pixels in a 4x4 range around the point to be interpolated, mainly by finding a coefficient to represent the influence factor of these 16 pixels on the pixel value at the point to be interpolated, and then calculate the pixel value of the point to be interpolated according to this influence factor, and its calculation formula is shown in Formula 1:
[0099]
[0100] In the above formula, (x, y) represents the coordinates of the point to be interpolated, (x i ,y i ) represents the coordinates of 16 points around the interpolation point, where i∈(0, 1, 2, 3), j∈(0, 1, 2, 3), f(x, y) represents the pixel value at the coordinate (x, y), and h(x) represents the weight coefficient at the distance x from the interpolation point. It is calculated using the bicubic function, h(x) = sin(πx) / (πx). The function is simplified as shown in Formula 2:
[0101]
[0102] In the embodiments of the present application, it is assumed that i∈(0, 1, 2, 3), and in the specific implementation, it can also be set to i∈(1, 2, 3, 4). Similarly, in the embodiments of the present application, it is assumed that j∈(0, 1, 2, 3), and in the specific implementation, it can also be set to j∈(1, 2, 3, 4), and this application does not make specific limitations.
[0103] See also Figure 5 , is a schematic diagram comparing the simplified function and bicubic function curves provided in the embodiment of the present application, wherein the solid line is the simplified function image, and the dotted line is the bicubic function image.
[0104] Then the simplified function is fixed-pointed to obtain the fixed-point function:
[0105]
[0106] In the above formula, h(x) represents the weight coefficient at a point x where the distance from the point to be interpolated is, and m represents the bit width occupied by the decimal part after fixed-point conversion.
[0107] Generally, when performing bicubic interpolation, 16 pixel points within a 4x4 range around the point to be interpolated are used to calculate the pixel value of the point to be interpolated, and 36 multiplication operations are required to calculate the interpolation result, which is relatively large in amount of calculation. Therefore, an embodiment of the present application provides a bicubic interpolation method in which a device first performs horizontal interpolation on an FPGA and then performs vertical interpolation on the result of the horizontal interpolation, and only 24 multiplication operations are required.
[0108] When performing horizontal interpolation, take 4 pixels in the horizontal direction, and the calculation formula is: Where: x represents the horizontal coordinate of the point to be interpolated, x i represents the horizontal coordinate of the i-th point among the four points around the interpolation point, where i∈(0, 1, 2, 3), f(x i ) represents the horizontal coordinate x i The pixel value of the image to be registered at the horizontal coordinate x is represented by h(x), and g(x) represents the pixel value after horizontal interpolation at the horizontal coordinate x. i ) can represent the weight of the i-th pixel value and can be obtained by the above formula (3).
[0109] It should be understood that, among the four pixel points in the image to be registered, the first pixel point can be understood as the point with the smallest horizontal coordinate value, and the fourth pixel point can be understood as the point with the largest horizontal coordinate value. In the above formulas (1) to (3), the first pixel point can be considered as i=0, the second pixel point can be considered as i=1, the third pixel point can be considered as i=2, and the fourth pixel point can be considered as i=3.
[0110] When the device implements interpolation calculation on FPGA, it first reads all pixel data of the image to be registered from the Double Data Rate Synchronous Dynamic Random Access Memory (DDR) and writes it into the Random Access Memory (RAM) RAM0, where the size of RAM0 is twice the number of pixels of the image to be registered. In this way, ping-pong operation can be performed when reading and writing RAM0.
[0111] The steps of the ping-pong operation are as follows: in the first buffer cycle, the input data stream is cached to the first data buffer module; in the second buffer cycle, the input data stream is cached to the second data buffer module through the switching of the input data selection unit, and the first cycle data cached by the first data buffer module is sent to the data stream operation processing module for operation processing through the selection of the input data selection unit; in the third buffer cycle, the input data stream is cached to the first data buffer module through the switching of the input data selection unit again, and the second cycle data cached by the second data buffer module is sent to the data stream operation processing module for operation processing through the switching of the input data selection unit, and so on. The ping-pong operation can make the data pass through the input data selection unit and the output data selection unit in a beat and coordinated switching, and the buffered data stream is sent to the data stream operation processing module for operation and processing without pause, and the buffer space can also be saved.
[0112] Then read all the pixel data of the image to be registered from RAM0, and write them into RAM1_0, RAM1_1, RAM1_2 and RAM1_3 in columns according to their positions in the image to be registered, wherein the size of these four RAMs is one-fourth of RAM0, and the pixel points in the 4mth column are stored in RAM1_0, the pixel points in the 4m+1th column are stored in RAM1_1, the pixel points in the 4m+2th column are stored in RAM1_2, and the pixel points in the 4m+3th column are stored in RAM1_3, where m is an integer greater than or equal to 0. Storing pixel points in this way can ensure that the 4 adjacent numbers in each row of data are respectively stored in 4 RAMs, so that the required 4 pixel points can be obtained at one time when performing horizontal interpolation. Then take out the required pixel points from RAM1_0, RAM1_1, RAM1_2, and RAM1_3 according to the coordinate values of the first coordinate transformation table.
[0113] Based on the above solution, since the RAM can only fetch one data at a time, by storing the pixels in four RAMs by columns, the requirement of fetching four data at a time in the bicubic interpolation algorithm can be met to complete horizontal interpolation.
[0114] The device performs horizontal interpolation calculation on the FPGA according to the four pixel points taken out in S202, obtains the first pixel value (interpolation calculation result), and then stores the interpolation calculation result in RAM2 by row. The size of RAM2 is three times the number of pixel points in a row in the image to be registered, and is used to cache three rows of pixel points.
[0115] S204, the device obtains four first pixel values corresponding to the nth element from N first pixel values through FPGA.
[0116] When performing interpolation in the vertical direction, the four first pixel values corresponding to the nth element can be taken out from RAM2. Among them, the column number of the four first pixel values is the same as the column number of the nth element, and the row number of the four first pixel values is adjacent. Since the first pixel value is calculated according to the order of the elements, the N first pixel values calculated in S203 are stored according to the position of the elements, and the row number and column number of the first pixel value represent the relative position of the first pixel value in the N first pixel values. For example, if a first pixel value is located in the 2nd row and the 3rd column of the N first pixel values, the row number of the first pixel value is 2 and the column number is 3. Assuming that the nth element is located in the 2nd column and the 3rd row of the first coordinate transformation table, the four pixel values are respectively the first pixel value of the 2nd column and the 3rd row of the N first pixel values, the first pixel value of the 2nd column and the 4th row of the N first pixel values, the first pixel value of the 2nd column and the 5th row of the N first pixel values, and the first pixel value of the 2nd column and the 6th row of the N first pixel values.
[0117] S205, the device calculates the second pixel value through FPGA according to the four first pixel values and the vertical coordinates of the elements in the first coordinate transformation table corresponding to the four first pixel values respectively.
[0118] The second pixel value can be considered as an interpolation result obtained by vertically interpolating the above four first pixel values. Specifically, when performing vertical interpolation, the calculation formula is: Where: y represents the ordinate of the point to be interpolated, g(y i ) represents the i-th first pixel value among the above four first pixel values, where i∈(0, 1, 2, 3), and p(y) represents the final pixel value at the vertical coordinate y.
[0119] Among them, h(yy i ) is obtained by the above formula (3). Since the row numbers of these four pixel values are adjacent, the distance between two pixel values can be considered to be 1. Therefore, after obtaining the four first pixel values, h(xx) can be calculated according to the decimal part of the ordinate of the nth element. i ). Among them, h(xx i ) can be calculated by the above formula (3), but substituting (xx in the formula i ) is calculated based on the decimal part of the ordinate of the nth element, where i can represent the i-th first pixel value among the above four first pixel values. Among the above four first pixel values, the first pixel value can be considered as i=0, the second pixel value can be considered as i=1, and so on. Specifically, the first pixel value (xx 0 )=|the fractional part of the ordinate of the nth element+1|, the second pixel value (xx 1)=|fractional part of the ordinate of the nth element|, (x 2 )=|1-the fractional part of the ordinate of the nth element|, the fourth pixel value (xx 3 ) = |2-the fractional part of the ordinate of the nth element|.
[0120] Based on the above scheme, when interpolation is performed in FPGA, the image to be registered is firstly interpolated horizontally, and then interpolated vertically based on the horizontal interpolation. This can reduce the use of resources, especially the use of multipliers, while also ensuring the quality of image registration.
[0121] In a possible implementation, the above-mentioned horizontal interpolation and vertical interpolation can also be performed in parallel. Specifically, the device can execute the above-mentioned S202 and S203 through the FPGA to obtain the horizontal interpolation results (first pixel values) of the elements in the first three rows of the first coordinate transformation table. After the device executes the above-mentioned S202 and S203 through the FPGA to obtain the horizontal interpolation results of the elements in the fourth row and first column of the first coordinate transformation table, the device can execute the above-mentioned S204 and S205 through the FPGA to obtain the second pixel value of the first element in the first coordinate transformation table. At this time, the device can obtain the first pixel values of the remaining elements in the first coordinate transformation table through the above-mentioned S202 and S203 of the FPGA, and simultaneously obtain the second pixel values of the remaining elements in the first coordinate transformation table by executing the above-mentioned S204 and S205 through the FPGA.
[0122] See also Figure 6 , is a structural diagram of row caching of horizontal interpolation results provided in an embodiment of the present application. Data, that is, the horizontal interpolation results, are input from the left end and output from the right end. Row cache 0, row cache 1, and row cache 2 correspond to the three rows of cached horizontal interpolation results, respectively. The output of row cache 0 will be input into row cache 1, and the output of row cache 1 will be input into row cache 2. The three row caches are all included in RAM2.
[0123] For example, after the device obtains the horizontal interpolation result of the first row, it inputs it into row buffer 0. When the horizontal interpolation result of the second row is obtained, the horizontal interpolation result of the first row is output from row buffer 0 and input into row buffer 1, and the horizontal interpolation result of the second row is stored in row buffer 0. When the horizontal interpolation result of the third row is obtained, the horizontal interpolation result of the first row is output from row buffer 1 and input into row buffer 2, the horizontal interpolation result of the second row is output from row buffer 0 and input into row buffer 1, and the interpolation result of the third row is input into row buffer 0. Then, when the device calculates and obtains the horizontal interpolation result of the first column of the fourth row, the horizontal interpolation result of the first row in row buffer 2, the horizontal interpolation result of the first column of the second row in row buffer 1, the horizontal interpolation result of the first column of the third row in row buffer 0, and the horizontal interpolation result of the first column of the fourth row just calculated are obtained from the output end at the same time, and these four horizontal interpolation results are output in sequence from the first column to the last column, and these four horizontal interpolation results are the four first pixel values used for vertical interpolation. A vertical interpolation result, that is, a second pixel value, is obtained by performing vertical interpolation calculation according to the four first pixel values. The method for calculating the second pixel value can refer to the relevant description in S205.
[0124] In a possible implementation, after obtaining the above vertical interpolation result, the registration of the image to be registered is realized. The device can perform cross-clock domain operation on the image registration result through a first-in first-out memory (FIFO), so that its clock frequency is the same as the clock frequency of the DDR, and then read it from the FIFO and store it in the DDR.
[0125] It should be noted that when performing vertical interpolation, when calculating the vertical interpolation of the first row and the last two rows of elements in the first coordinate transformation table, the problem of insufficient first pixel value will occur. Therefore, when the device performs the vertical interpolation result of the first row of elements through FPGA, it can perform a padding operation on the first pixel value of the first row of elements. At this time, the row number of the first pixel value of the first row of elements can be regarded as 2, and the row number of the first pixel value obtained by the padding operation can be regarded as 1. Also, when the device calculates the vertical interpolation result of the last two rows of elements through FPGA, it performs a padding operation below the first pixel value of the last two rows to complete the vertical interpolation of the last two rows of elements in the first coordinate transformation table. Among them, this application does not make specific restrictions on how to perform the padding operation, and it is sufficient to complete the vertical interpolation of the first row of elements and the last two rows of elements in the first coordinate transformation table.
[0126] The technical solution provided by this application is described below through specific embodiments. Figure 6 , which is one of the exemplary flow charts of an image registration method provided in an embodiment of the present application.
[0127] S701, the device obtains a second coordinate transformation table of the target image after performing feature extraction and feature matching on the CPU.
[0128] The method for obtaining the second coordinate transformation table on the CPU can be found in Figure 2 The relevant descriptions in the method embodiment shown are not repeated here.
[0129] S702, the device downsamples the second coordinate transformation table on the CPU to obtain a third coordinate transformation table and sends it to the FPGA.
[0130] The method for downsampling the second coordinate transformation table on the CPU can be found in Figure 2 The relevant descriptions in the method embodiment shown are not repeated here.
[0131] S703, the device upsamples the third coordinate transformation table on the FPGA to obtain a first coordinate transformation table.
[0132] The method for upsampling the third coordinate transformation table on the FPGA can be found in Figure 2 The relevant descriptions in the method embodiment shown are not repeated here.
[0133] S704, the device reads all pixel data of the image to be registered from DDR through FPGA and writes them into RAM0.
[0134] S705, the device reads all pixel point data of the image to be registered in RAM0 through FPGA and writes them into RAM1_0, RAM1_1, RAM1_2 and RAM1_3 in columns.
[0135] S706, the device reads pixel points from RAM1_0, RAM1_1, RAM1_2 and RAM1_3 on the FPGA according to the coordinate values and calculates horizontal interpolation.
[0136] The device reads pixel points from RAM1_0, RAM1_1, RAM1_2 and RAM1_3 on the FPGA according to the coordinate values on the first coordinate transformation table obtained in S703 and calculates horizontal interpolation. The method of horizontal interpolation can be found in FIG. Figure 2 The relevant descriptions in the method embodiment shown are not repeated here.
[0137] S707, the device stores the horizontal interpolation result into RAM2 through FPGA, and caches three rows of pixels.
[0138] S708, the device reads data from RAM2 through FPGA and performs vertical interpolation.
[0139] The method of vertical interpolation can be found in Figure 2 The relevant descriptions in the method embodiment shown are not repeated here.
[0140] S709, the device stores the vertical interpolation result into FIFO through FPGA to cross the clock domain.
[0141] After completing the interpolation calculation in S708, the interpolation result in the vertical direction is stored in the FIFO for cross-clock domain operation, so that its clock frequency is the same as the clock frequency of the DDR, so as to facilitate the subsequent writing of the image registration result into the DDR.
[0142] S710, the device reads the image registration result from the FIFO through the FPGA and stores it in the DDR.
[0143] The registration results are stored in DDR for easy reading by the device.
[0144] Based on the same concept of the above method, see Figure 8 , is an image registration device 800 provided in an embodiment of the present application, the device 800 can perform each step in the above method, and in order to avoid repetition, it will not be described in detail here. The device 800 includes a storage unit 801 and a processing unit 802.
[0145] In one scenario:
[0146] The storage unit is used to store computer programs or instructions;
[0147] The processing unit is used to perform the following operations:
[0148] A first coordinate transformation table of the target image is obtained through a field programmable gate array FPGA; the first coordinate transformation table includes N elements, where N is a positive integer; wherein one element is the coordinate position of a pixel point in the target image in the image to be registered; the target image and the image to be registered are images of the same object, and the generation conditions of the target image and the image to be registered are different;
[0149] Acquire four pixel points in the image to be registered according to the nth element through the FPGA; the nth element is any element in the first coordinate transformation table, and n is any integer between 1 and N in turn; the four pixel points in the image to be registered are pixel points in the same row of the image to be registered; the row where the four pixel points are located is determined according to the ordinate of the nth element;
[0150] Calculating, by the FPGA, a first pixel value of a pixel point corresponding to an nth element in the image to be registered according to the four pixel points, where n is any integer between 1 and N in sequence;
[0151] Obtaining four first pixel values corresponding to the nth element from the N first pixel values by the FPGA; the column numbers of the four first pixel values are the same as the column number of the nth element, the row numbers of the four first pixel values are adjacent, and the row number of the first first pixel value among the four first pixel values is the same as the row number of the nth element;
[0152] The second pixel value is calculated by the FPGA according to the four first pixel values and the ordinate of the nth element.
[0153] In a possible implementation, the step of acquiring a first coordinate transformation table of a target image through an FPGA includes:
[0154] Extracting features from the image to be registered and the target image respectively by using a processor CPU to obtain a first feature point set of the image to be registered and a second feature point set of the target image;
[0155] Determining a feature point pair set by the CPU according to a similarity measure; the feature point pair set includes a plurality of feature point pairs, one feature point pair includes a first feature point in the first feature point set and a second feature point in the second feature point set; a similarity measure between the first feature point and the second feature point is greater than or equal to a first threshold;
[0156] The first coordinate transformation table is determined by the CPU according to the set of feature point pairs.
[0157] In a possible implementation manner, determining the first coordinate transformation table according to the feature point pair set includes:
[0158] Determining, by the CPU, a second coordinate transformation table according to the set of feature point pairs;
[0159] The CPU performs k-fold downsampling on the second coordinate transformation table to obtain a third coordinate transformation table; the number of elements in the second coordinate transformation table is k times the number of elements in the third coordinate transformation table; k is an integer greater than 1;
[0160] The third coordinate transformation table is upsampled by k times through the FPGA to obtain the first coordinate transformation table; the number of elements in the first coordinate transformation table is the same as the number of elements in the second coordinate transformation table.
[0161] In a possible implementation, acquiring four pixel points in the image to be registered according to the horizontal coordinate of the nth element by the FPGA includes:
[0162] The FPGA is used to obtain four pixel points from four RAMs according to the horizontal coordinate of the nth element; wherein one pixel point is obtained from each RAM; the four RAMs respectively store partial pixel points of the image to be registered; and the number of pixel points stored in the four RAMs is the same.
[0163] In one possible implementation, the 4mth column of pixels in the image to be registered are stored in the first RAM, the 4m+1th column of pixels are stored in the second RAM, the 4m+2th column of pixels are stored in the third RAM, and the 4m+3th column of pixels are stored in the fourth RAM, where m is an integer greater than or equal to 0.
[0164] In a possible implementation manner, it is characterized in that after calculating the first pixel value corresponding to the horizontal coordinate of the nth element according to the four pixel points by the FPGA, it further includes:
[0165] Cache the obtained N first pixel values into a fifth RAM;
[0166] The acquiring, by the FPGA according to the ordinate of the nth element, four first pixel values among the N first pixel values comprises:
[0167] The FPGA is used to obtain four first pixel values among the N first pixel values from the fifth RAM according to the vertical coordinate of the nth element.
[0168] In a possible implementation, after the processing unit calculates the second pixel value corresponding to the ordinate of the nth element according to the four first pixel values through the FPGA, the processing further includes:
[0169] Performing a cross-clock domain operation on the N second pixel values through a first-in first-out memory FIFO; wherein the clock frequency of the N second pixel values after the cross-clock domain operation is the same as the clock frequency of a double data rate synchronous dynamic random access memory DDR;
[0170] The N second pixel values after the cross-clock domain operation are stored in DDR.
[0171] Based on the same concept of the above method, see Fig. 9 , is an electronic device provided in an embodiment of the present application, the electronic device includes a processor 901 and a memory 902. The memory 902 is used to store computer-executable instructions, and the processor 901 executes the computer-executable instructions in the memory to use the hardware resources in the controller to perform the operation steps of the method in any possible implementation of the above method. The processor 901 can be used to execute the operation of the processing unit 802, and the memory 902 can be used to execute the operation of the storage unit 801.
[0172] An embodiment of the present application also provides a computer storable medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.
[0173] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0174] Although the specific embodiments of the present application are described above, it should be understood by those skilled in the art that these are merely examples, and the scope of protection of the present application is limited by the attached claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present application, but these changes and modifications all fall within the scope of protection of the present application. Although the preferred embodiments of the present application have been described, once those skilled in the art are aware of the basic creative concepts, additional changes and modifications may be made to these embodiments. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0175] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An image registration method, characterized in that: include: Acquire a first coordinate transformation table of the target image through a field programmable gate array FPGA; The first coordinate transformation table includes N elements, where N is a positive integer; wherein one element is the coordinate position of a pixel point in the target image in the image to be registered; the target image and the image to be registered are images of the same object, and the generation conditions of the target image and the image to be registered are different; Acquire four pixel points in the image to be registered according to the nth element through the FPGA; the nth element is any element in the first coordinate transformation table, and n is any integer between 1 and N in turn; the four pixel points in the image to be registered are pixel points in the same row of the image to be registered; the row where the four pixel points are located is determined according to the ordinate of the nth element; Calculating, by the FPGA, a first pixel value of a pixel point corresponding to the nth element in the image to be registered according to the four pixel points and the horizontal coordinate of the nth element, where n is any integer between 1 and N; Obtaining four first pixel values corresponding to the nth element from the N first pixel values by the FPGA; the column numbers of the four first pixel values are the same as the column number of the nth element, the row numbers of the four first pixel values are adjacent, and the row number of the first first pixel value among the four first pixel values is the same as the row number of the nth element; The second pixel value is calculated by the FPGA according to the four first pixel values and the ordinate of the nth element.
2. The method according to claim 1, characterized in that The method of obtaining a first coordinate transformation table of a target image through FPGA includes: Extracting features from the image to be registered and the target image respectively by using a processor CPU to obtain a first feature point set of the image to be registered and a second feature point set of the target image; Determining a feature point pair set by the CPU according to a similarity measure; the feature point pair set includes a plurality of feature point pairs, one feature point pair includes a first feature point in the first feature point set and a second feature point in the second feature point set; a similarity measure between the first feature point and the second feature point is greater than or equal to a first threshold; The first coordinate transformation table is determined by the CPU according to the set of feature point pairs.
3. The method according to claim 2, characterized in that Determining the first coordinate transformation table according to the feature point pair set includes: Determining, by the CPU, a second coordinate transformation table according to the set of feature point pairs; The CPU performs k-fold downsampling on the second coordinate transformation table to obtain a third coordinate transformation table; the number of elements in the second coordinate transformation table is k times the number of elements in the third coordinate transformation table; k is an integer greater than 1; The third coordinate transformation table is upsampled by k times through the FPGA to obtain the first coordinate transformation table; the number of elements in the first coordinate transformation table is the same as the number of elements in the second coordinate transformation table.
4. The method according to any one of claims 1 to 3, characterized in that: The acquiring four pixel points in the image to be registered according to the nth element by the FPGA includes: The FPGA is used to obtain four pixel points from four RAMs according to the horizontal coordinate of the nth element; wherein one pixel point is obtained from each RAM; the four RAMs respectively store partial pixel points of the image to be registered; and the number of pixel points stored in the four RAMs is the same.
5. The method according to claim 4, characterized in that The 4mth column of pixels in the image to be registered is stored in the first RAM, the 4m+1th column of pixels is stored in the second RAM, the 4m+2th column of pixels is stored in the third RAM, and the 4m+3th column of pixels is stored in the fourth RAM, where m is an integer greater than or equal to 0.
6. The method according to any one of claims 1 to 3 or 5, characterized in that: After calculating the first pixel value of the pixel point corresponding to the nth element in the image to be registered according to the four pixel points and the horizontal coordinate of the nth element by the FPGA, the method further includes: Cache the obtained N first pixel values into a fifth RAM; The acquiring, by the FPGA according to the ordinate of the nth element, four first pixel values among the N first pixel values comprises: The FPGA is used to obtain four first pixel values among the N first pixel values from the fifth RAM according to the vertical coordinate of the nth element.
7. The method according to any one of claims 1 to 3 or 5, characterized in that: After calculating the second pixel value according to the four first pixel values and the ordinate of the nth element by the FPGA, the method further includes: Performing a cross-clock domain operation on the N second pixel values through a first-in first-out memory FIFO; wherein the clock frequency of the N second pixel values after the cross-clock domain operation is the same as the clock frequency of a double data rate synchronous dynamic random access memory DDR; The N second pixel values after the cross-clock domain operation are stored in DDR.
8. An image registration device, characterized in that: Including storage unit and processing unit: The storage unit is used to store computer programs or instructions; The processing unit is used to perform the following operations: A first coordinate transformation table of the target image is obtained through a field programmable gate array FPGA; the first coordinate transformation table includes N elements, where N is a positive integer; wherein one element is the coordinate position of a pixel point in the target image in the image to be registered; the target image and the image to be registered are images of the same object, and the generation conditions of the target image and the image to be registered are different; Acquire four pixel points in the image to be registered according to the nth element through the FPGA; the nth element is any element in the first coordinate transformation table, and n is any integer between 1 and N in sequence; the four pixel points in the image to be registered are pixel points in the same row of the image to be registered; Calculating, by the FPGA, a first pixel value of a pixel point corresponding to the nth element in the image to be registered according to the four pixel points and the horizontal coordinate of the nth element, where n is any integer between 1 and N; Obtaining four first pixel values corresponding to the nth element from the N first pixel values by the FPGA; the column numbers of the four first pixel values are the same as the column number of the nth element, the row numbers of the four first pixel values are adjacent, and the row number of the first first pixel value among the four first pixel values is the same as the row number of the nth element; The second pixel value is calculated by the FPGA according to the four first pixel values and the ordinate of the nth element.
9. The device according to claim 8, characterized in that The method of obtaining a first coordinate transformation table of a target image through FPGA includes: Extracting features from the image to be registered and the target image respectively by using a processor CPU to obtain a first feature point set of the image to be registered and a second feature point set of the target image; Determining a feature point pair set by the CPU according to a similarity measure; the feature point pair set includes a plurality of feature point pairs, one feature point pair includes a first feature point in the first feature point set and a second feature point in the second feature point set; a similarity measure between the first feature point and the second feature point is greater than or equal to a first threshold; The first coordinate transformation table is determined by the CPU according to the set of feature point pairs.
10. The device according to claim 9, characterized in that Determining the first coordinate transformation table according to the feature point pair set includes: Determining, by the CPU, a second coordinate transformation table according to the set of feature point pairs; The CPU performs k-fold downsampling on the second coordinate transformation table to obtain a third coordinate transformation table; the number of elements in the second coordinate transformation table is k times the number of elements in the third coordinate transformation table; k is an integer greater than 1; The third coordinate transformation table is upsampled by k times through the FPGA to obtain the first coordinate transformation table; the number of elements in the first coordinate transformation table is the same as the number of elements in the second coordinate transformation table.
11. The device according to any one of claims 8 to 10, characterized in that: The acquiring four pixel points in the image to be registered according to the nth element by the FPGA includes: The FPGA is used to obtain four pixel points from four RAMs according to the horizontal coordinate of the nth element; wherein one pixel point is obtained from each RAM; the four RAMs respectively store partial pixel points of the image to be registered; and the number of pixel points stored in the four RAMs is the same.
12. The device according to claim 11, characterized in that The 4mth column of pixels in the image to be registered is stored in the first RAM, the 4m+1th column of pixels is stored in the second RAM, the 4m+2th column of pixels is stored in the third RAM, and the 4m+3th column of pixels is stored in the fourth RAM, where m is an integer greater than or equal to 0.
13. The device according to any one of claims 8 to 10 or 12, characterized in that: After calculating the first pixel value of the pixel point corresponding to the nth element in the image to be registered according to the four pixel points and the horizontal coordinate of the nth element by the FPGA, the method further includes: Cache the obtained N first pixel values into a fifth RAM; The acquiring, by the FPGA according to the ordinate of the nth element, four first pixel values among the N first pixel values comprises: The FPGA is used to obtain four first pixel values among the N first pixel values from the fifth RAM according to the vertical coordinate of the nth element.
14. The device according to any one of claims 8 to 10 or 12, characterized in that: After calculating the second pixel value according to the four first pixel values and the ordinate of the nth element by the FPGA, the method further includes: Performing a cross-clock domain operation on the N second pixel values through a first-in first-out memory FIFO; wherein the clock frequency of the N second pixel values after the cross-clock domain operation is the same as the clock frequency of a double data rate synchronous dynamic random access memory DDR; The N second pixel values after the cross-clock domain operation are stored in DDR.
15. An electronic device, characterized in that: The electronic device comprises a processor and a memory, The memory is used to store computer programs or instructions; The processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 7 is executed.
16. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
