A multi-frame star map superposition FPGA implementation method and system

Abstract

The application discloses a star map attitude correlation parallel superposition method and system based on an FPGA pipeline cache and a multi-channel DDR architecture, and belongs to the field of astronomical navigation image enhancement. In view of the problems of long data carrying time and insufficient computing power when the existing ARM PS end processes star map superposition, the application implements algorithms such as attitude correlation coordinate conversion, area weighted projection, in-frame fusion and multi-frame starting coordinate alignment on the PL end hardware of the FPGA, and the PS end only provides corner point coordinate parameters. Coordinate conversion is realized through fixed-point optimization pipeline, overlapping pixel alignment fusion is completed by using FIFO row cache, and superposition is performed on multiple images read in parallel by a multi-channel DDR controller. The application realizes data 'in-situ calculation', avoids frequent intervention of the ARM, improves the processing speed by tens of times, meanwhile, the consistency of the floating-point operation precision is ensured, and is suitable for star map signal-to-noise ratio enhancement under high dynamic and strong noise.

Description

Technical Field

[0001] This invention relates to the field of embedded FPGA image processing technology, specifically to a star sensor image processing technology based on FPGA (Field Programmable Gate Array) hardware acceleration, and in particular to a method and system for realizing star map attitude correlation transformation, intra-frame image fusion and multi-frame parallel overlay using inertial navigation information assistance and based on pipelined buffer and multi-channel DDR architecture. Background Technology

[0002] In inertial / astronomical integrated navigation systems, especially in daylight or high-dynamic, noisy environments, the signal-to-noise ratio of a single-frame star image is often low, making it difficult to meet the requirements for star identification. To improve detector sensitivity, multi-frame stacking techniques are typically used to enhance the signal. However, because the movement of the carrier causes the stars to shift on the imaging focal plane, direct stacking will produce trailing blur.

[0003] In existing technologies, such as the "attitude-correlated frame superposition method" proposed by Ni Yuanman of the National University of Defense Technology in "Research on Inertial / Astronomical Deep Combined Star Map Enhancement and Error Suppression Technology," this method utilizes attitude information provided by the Inertial Measurement Unit (IMU) to dynamically correlate multiple frames of images to the same reference coordinate system at the same moment through coordinate transformation, and then performs area-weighted superposition, thereby enhancing the signal-to-noise ratio while maintaining the star point shape. Another example is Dai Kaidong's "A Star Map Signal-to-Noise Ratio Enhancement Method Based on Attitude-Correlated Frame Superposition," which proposes a method to achieve multi-frame star map correlation superposition by combining gyroscope measurement data. This method fixes the star sensor to a three-axis gyroscope unit (GUs), using the GUs to accurately measure the attitude change information of two star maps taken by the star sensor at different times. Based on this attitude change, the star map is translated and rotated, thereby achieving dynamic star map correlation superposition. However, this algorithm involves complex floating-point coordinate transformations, sub-pixel interpolation, and frequent data transfer between multiple frames.

[0004] Current engineering implementations typically employ an architecture of "Sensor -> FPGA -> DDR -> ARM PS (IMU) -> DDR -> PFGA". That is, the FPGA is responsible for image acquisition, the inertial navigation system (INS) provides attitude information, and the ARM PS (processing system) reads the image data from the DDR and performs algorithmic processing in conjunction with the attitude information provided by the INS. This approach has the following drawbacks: 1. Data transfer is time-consuming: The ARM side needs to frequently read and write image data in DDR. Due to bandwidth limitations, the processing latency is large, making it difficult to meet the real-time requirements of high frame rate.

[0005] 2. Processing capacity bottleneck: The pose correlation algorithm involves a large number of matrix operations and interpolation calculations. The ARM processor has insufficient computing power when processing multi-frame high-resolution images, which limits the system response speed.

[0006] 3. Waste of system resources: Data is moved back and forth between the FPGA and ARM, which consumes AXI bus bandwidth.

[0007] Therefore, there is an urgent need for a hardware acceleration architecture that can complete the entire process on the FPGA to solve the speed bottleneck of the above algorithm implementation. Summary of the Invention

[0008] This invention aims to overcome the shortcomings of existing ARM PS-side processing of star map overlay, such as time-consuming data transfer and poor real-time performance. It provides a parallel overlay system and method for star map attitude association based on FPGA pipelined cache and multi-channel DDR architecture. By implementing all core algorithms in hardware on the FPGA PL side, it achieves "in-situ calculation" of data and parallel overlay of multiple frames, significantly improving processing speed and real-time performance.

[0009] In a first aspect, the present invention provides a star map attitude correlation parallel overlay system based on FPGA pipelined cache and multi-channel DDR architecture, comprising: The attitude-related coordinate transformation module is used to receive the coordinates of the four corner points of the current frame image calculated and sent by the PS end based on inertial navigation information, and generate the mapped coordinates of each pixel in the current frame image to the reference coordinate system through pipeline operation according to the relative position of each pixel in the image. An area-weighted projection calculation module is used to receive the mapped coordinates and calculate the area-weighted weight value of each pixel in the current frame image projected onto the adjacent integer pixels in the reference coordinate system based on the overlapping area between the mapped coordinates and the integer pixel coordinates in the reference coordinate system. The image intra-frame fusion module is used to read the original pixel values ​​of the current frame image from DDR, perform grayscale allocation according to the area weighting value, and then perform time-series alignment and superposition of the allocated grayscale values ​​through the line buffer pipeline to generate the intra-frame fused image of the current frame image in the reference coordinate system. The multi-frame image start coordinate alignment module is used to receive the start coordinates of the intra-frame fused images and determine the start coordinates of the overlapping area of ​​all the frames to be superimposed through comparison calculations. The multi-channel DDR read / write control module is used to control the transfer and scheduling of image data between the DDR and various modules, and based on the starting coordinates of the overlapping area, reads the overlapping area data of multiple frames of the intra-frame fused image from the DDR in parallel through multiple channels, and performs the final multi-frame image overlay.

[0010] In some instances, the attitude-associated coordinate transformation module includes: The pixel counter and the row counter are used to count the horizontal and vertical numbers of the currently processed pixel within the frame, respectively. The first ROM and the second ROM use the horizontal sequence number and the vertical sequence number as read addresses respectively, and store the pre-calculated horizontal pixel offset ratio and vertical pixel offset ratio accordingly; The pipelined multiplication and addition unit is used to perform hierarchical multiplication and addition operations on the horizontal pixel offset ratio, the vertical pixel offset ratio and the coordinates of the four corner points, and output the mapped coordinates of each pixel in a pipelined manner.

[0011] In some instances, the area-weighted projection calculation module is specifically used for: The decimal coordinate values ​​of the mapped coordinates are rounded down and up to obtain four adjacent integer pixel coordinates in the reference coordinate system. The validity of the four adjacent integer pixel coordinates is determined, and the out-of-bounds coordinates are clamped. Based on the relative distance between the mapped coordinates and the coordinates of the four adjacent integer pixels, the area weighting value corresponding to the mapped coordinates for the four adjacent integer pixels is calculated by a multiply-accumulate pipeline.

[0012] In some instances, the intra-frame image fusion module includes: A timing alignment unit is used to synchronize the area weighting value with the corresponding original pixel value read from DDR. The grayscale allocation operation unit is used to multiply the original pixel value with the corresponding area weighting value to obtain the grayscale components allocated to four adjacent integer pixel positions in the reference coordinate system. A row buffer pipeline unit includes at least one FIFO or row buffer for caching data belonging to different rows in the grayscale component, so as to achieve temporal alignment of overlapping pixels within the same row and between adjacent rows. The pixel overlay unit is used to accumulate all grayscale components at the four adjacent integer pixel positions after alignment to generate the pixel value of the intra-frame fused image.

[0013] In some instances, the multi-frame image start coordinate alignment module includes: The shift register group is used to stitch together and store the starting horizontal and starting vertical coordinates of the intra-frame fused images of consecutive input frames; A multi-level comparator is used to compare multiple initial horizontal and initial vertical coordinates after splicing, and select the maximum value as the starting coordinate of the overlapping area; The address calculation unit is used to calculate the read base address of the intra-frame fused image in DDR based on the starting coordinates of the overlapping region and the preset superimposed image size.

[0014] In some instances, the multi-channel DDR read / write control module includes: The first DDR control channel is connected to a high-speed data interface of the FPGA, used to write the raw image data acquired by the image sensor into the DDR, and to read the raw image data according to the request of the image intra-frame fusion module. The second DDR control channel is connected to another high-speed data interface of the FPGA and contains multiple parallel read control sub-channels. Each sub-channel is used to read the overlapping region data of an intra-frame fused image in parallel according to the read base address of the corresponding frame calculated by the address calculation unit.

[0015] In some instances, the system uses the FPGA's pixel clock as the operating clock for all algorithm processing modules, and the AXI bus clock as the clock for parameter interaction between the PS terminal and the FPGA. The PS terminal and the FPGA transmit corner coordinates and control parameters through a low-speed bus interface and transmit image data through a high-speed data interface.

[0016] Secondly, this invention provides a method for parallel overlay of star map attitude correlation based on FPGA pipelined cache and multi-channel DDR architecture, including: Coordinate transformation steps: The FPGA receives the coordinates of the four corner points of the current frame image calculated by the PS terminal based on the inertial navigation information and transforms them into the reference coordinate system. The FPGA generates the mapped coordinates of each pixel through pipeline operation based on the corner point coordinates and the positional relationship of each pixel in the image. Weighted projection step: The FPGA calculates the area weighting value of each pixel projected onto the adjacent integer pixels based on the overlapping area between the mapped coordinates and the integer pixel coordinates in the reference coordinate system; Intra-frame fusion steps: The FPGA reads the original pixel values ​​of the current frame image from the DDR, allocates grayscale according to the area weighting value, and performs timing alignment and overlay through the line buffer pipeline to generate an intra-frame fused image; Starting coordinate alignment step: The FPGA receives the starting coordinates of the intra-frame fused images of each frame and determines the starting coordinates of the overlapping area of ​​all frames to be superimposed through comparison calculation; Multi-frame parallel overlay step: The FPGA reads the overlapping region data of multiple frames of intra-frame fused images in parallel based on the starting coordinates of the overlapping region through a multi-channel DDR controller, and performs the final overlay.

[0017] In some instances, the intra-frame fusion step specifically includes: The area-weighted values ​​are synchronized with the original pixel values ​​read from the DDR in a timing manner; Multiply the synchronized original pixel value by the corresponding weight to obtain the grayscale component; FIFO or row buffering is used to cache different rows of data in the grayscale component to achieve temporal alignment of overlapping pixels within and between rows; All aligned grayscale components are summed to output the pixel values ​​of the intra-frame fused image.

[0018] In some instances, the multi-frame parallel overlay step is followed by: The cropping step involves cropping the common area image defined by the starting coordinates of the overlapping area and the preset overlay size from the final superimposed image, and outputting it as the effective overlay result.

[0019] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: This invention integrates attitude-related coordinate transformation, area-weighted projection, intra-frame image fusion, multi-frame start coordinate alignment, and multi-channel DDR read / write control into the FPGA's PL terminal in a hardware pipelined manner. The PS terminal is only responsible for corner coordinate parameter calculation and low-speed transmission. The entire process of image data acquisition, buffering, fusion, and final overlay requires no ARM intervention in DDR read / write operations, thus completely eliminating the AXI bus congestion and bandwidth bottlenecks caused by frequent data movement in existing technologies, achieving in-situ computation. Simultaneously, it fully utilizes the pipelined parallel processing capabilities of the FPGA, in conjunction with a multi-channel DDR controller, to achieve parallel reading and row processing of multi-frame fused images. The overlapping pixel alignment and stacking of the cached FIFO enables complex operations such as coordinate mapping, grayscale weighted allocation, and pixel stacking to be executed in parallel, improving processing speed by tens of times compared to the pure software ARM solution, meeting the real-time requirements of high frame rate. In addition, the computing power load on the PS side and the power consumption of DDR access are significantly reduced. Through the floating-point to fixed-point optimization design, the error between the hardware fixed-point operation result and the floating-point operation result is controlled within a very small range of grayscale level, ensuring a significant enhancement of the star map signal-to-noise ratio and the accuracy of subsequent star point recognition. The five core hardware modules have clear interfaces and strong scalability. The overall system has achieved comprehensive improvements in real-time performance, power consumption, accuracy, and engineering feasibility. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a schematic diagram of the pixel points after attitude correlation transformation provided in the embodiment of the present invention; Figure 2 This is a schematic diagram of the coordinate transformation pixel projection overlap area provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a parallel overlay system architecture based on FPGA pipelined cache and multi-channel DDR architecture star map attitude correlation provided in an embodiment of the present invention; Figure 4 This is a flowchart of the attitude-related coordinate transformation module provided in an embodiment of the present invention; Figure 5 This is a flowchart of the area-weighted projection calculation module provided in an embodiment of the present invention; Figure 6 This is a flowchart of the image intra-frame fusion module provided in an embodiment of the present invention; Figure 7 This is a flowchart of the multi-frame image start coordinate alignment module provided in an embodiment of the present invention; Figure 8 This is a flowchart of the multi-channel DDR read / write module provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In the following description, specific embodiments of the invention will be illustrated with reference to steps and symbols performed by one or more computers, unless otherwise stated. Therefore, these steps and operations will be referred to several times as being performed by a computer, and computer execution as referred to herein includes operations by a computer processing unit representing electronic signals of data in a structured format. This operation transforms the data or maintains it at a location in the computer's memory system, which can be reconfigured or otherwise alter the operation of the computer in a manner well known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of the invention described above are not intended to be limiting, and those skilled in the art will understand that many of the following steps and operations can also be implemented in hardware.

[0024] The terms "module" or "unit" as used herein can be considered as software objects executing on the computing system. Different components, modules, engines, and services described herein can be considered as implementations on the computing system. The apparatus and methods described herein are preferably implemented in software, but can also be implemented in hardware, both of which are within the scope of this invention.

[0025] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0026] This invention aims to improve the coordination and division of labor between the PS and PL ends of an embedded image processing system, and to achieve the following core functions through pure logic: 1. End-to-end hardware acceleration: The "image coordinate transformation", "area-weighted projection" and "multi-frame image overlay" algorithms, which were originally handled by the ARM PS side, are all ported to the FPGA PL side. The PS side is only responsible for using inertial navigation information to calculate the corner coordinates of the first n-1 frames of image transformation and providing them to the PL side for image coordinate transformation, thereby eliminating the data transfer bottleneck between the ARM and FPGA.

[0027] 2. Intra-frame image fusion: After the coordinates of the first n-1 frames are transformed to the nth frame, the pixels of each frame will overlap when mapped to the reference coordinate system. In order to reduce the number of times data is moved from DDR during image overlay, and to achieve parallel overlay of n frames, the overlapping pixels within the frame are aligned and overlaid by buffering the row data in the FIFO pipeline, thus achieving intra-frame image fusion and reducing redundant data movement.

[0028] 3. Multi-channel parallel processing: Utilizing the parallel computing characteristics of FPGA and the multi-channel DDR architecture, parallel reading and pipelined overlay of multiple frames of star map data are achieved, significantly improving processing speed.

[0029] 4. High precision and low error: Through fixed-point optimization design, the error between the FPGA hardware processing result and the PS terminal floating-point processing result is controlled within a very small gray level range, while ensuring a significant enhancement of the star map signal-to-noise ratio.

[0030] To achieve the above technical objectives, this invention provides a parallel overlay system for star map attitude correlation based on FPGA pipelined cache and multi-channel DDR architecture. The system architecture consists of the following five modules: 1) The attitude-related coordinate transformation module, based on the attitude information provided by the inertial navigation system and the image corner coordinates calculated by the PS terminal, informs the module of the coordinate positions of the four corner pixels after the current image is transformed into the reference coordinate system. This invention, based on the corner coordinate information of each frame, expands the horizontal and vertical pixel offset ratios using row and column counters, and combines floating-point and fixed-point optimization processing to calculate the coordinates of all pixels after transformation into the reference coordinate system, achieving calculation with minimal error accuracy and providing a data foundation for subsequent modules. The calculation process is performed in a pipelined structure, which disperses complex computational steps while unifying them overall, making the calculation process controllable and orderly.

[0031] 2) Area-weighted projection calculation module. This module mainly addresses the problem that after pose-related transformation, the coordinates contain decimal parts, and the pixels in the associated multi-frame images cannot completely overlap. It interpolates the non-integer transformed coordinates (u2, v2) of a pixel in the j-th frame image from module 1) to the four adjacent integer coordinate pixels (u3, v3), (u3, v4), (u4, v3), and (u4, v4) in the n-th frame image. The center of pixel (u2, v2) is A0, and it overlaps with four pixels in the n-th frame image. The centers of these four pixels are A1, A2, A3, and A4 respectively (e.g., ...). Figure 1 (As shown). Calculate the area projection ratio from the current pixel to four interpolated pixels. Divide the grayscale value of pixel A0 into four parts based on the size of the overlapping area, and then superimpose them onto the four pixels A1, A2, A3, and A4 of the nth frame image, respectively. The output area projection ratio is time-aligned with the original image data in the PS DDR cache.

[0032] 3) Image intra-frame fusion module: This module receives the area projection weights output by module 2), reads the original image data from the PS DDR buffer, performs grayscale allocation calculations, and then combines the row data pipeline FIFO buffer, output register snapshotting, and delay processing to achieve overlapping pixel alignment and fusion (as shown in the appendix). Figure 2 As shown in the figure, the number of times multi-frame superimposed image data is moved from DDR is reduced, and the complexity of the data transmission system is reduced, so as to realize the parallel superimposition of n frames of images.

[0033] 4) Multi-frame image start coordinate alignment module: The module receives the interpolated pixel coordinates output by module 2), determines whether they are the start coordinates of the fused image output by module 3) based on the row and column counter, splices the start coordinates of the four fused images through a shift register, optimizes the judgment timing through a multi-level comparator, unifies the start coordinates of the four fused images, and uses them to read out the overlapping areas of the four images in parallel for superposition processing.

[0034] 5) A multi-channel DDR read / write module, used for image data transfer and scheduling throughout the system. This module receives the original image acquired by the sensor, the intra-frame fused image output by module 3), and the image after multi-frame overlay. It then uses the pixel clock, data validity flag, and read / write address to transfer the image to the PS DDR, and reads the original image data from the PS DDR and outputs it to module 3) for image fusion calculation. Through the four-channel DDR read module and the starting address output by module 4), four fused images are read simultaneously to achieve multi-frame overlay. This invention utilizes FPGA logic resources to construct a high-efficiency DDR multi-channel read / write controller, ensuring instantaneous acquisition of multiple frames of data during overlay and enabling skip-reading of multi-frame data with discontinuous memory addresses.

[0035] This invention provides a parallel overlay system for star map attitude correlation based on FPGA pipeline caching and multi-channel DDR architecture, compared to an ARM PS-based full-process overlay algorithm. After the image is acquired by the sensor and enters the FPGA, it is directly cached by the multi-channel DDR controller and pipelined data processing is implemented. Data transfer is entirely independent of ARM intervention, solving the bus congestion caused by ping-pong operations and achieving "in-situ computation." Simultaneously, the efficient DDR multi-channel controller improves data bandwidth, and the dedicated hardware coordinate transformation pipeline enables real-time calculation of the sub-pixel mapping relationship between historical frames and the reference frame, fully utilizing the advantages of parallel processing. The processing speed is tens of times or more faster than that implemented on the PS side, avoiding data transfer bottlenecks. Global floating-point and fixed-point processing optimizes the fixed-point bit width, ensuring consistency between image quality and the high precision of floating-point operations after hardware acceleration.

[0036] like Figure 3 The diagram shown is an architectural schematic of a star map attitude correlation parallel overlay system based on an FPGA pipelined cache and multi-channel DDR architecture, provided by an embodiment of the present invention. The flowchart describes the data flow and the main functions of each module; detailed descriptions of each module will follow. It should be understood that... Figure 3 Each module displays only a portion of the core components; this does not mean that a unit only uses the display components.

[0037] The system takes into account raw image data from the image sensor, pixel clock, and frame synchronization signal. Outputs include the clock signal driving the image sensor via the PL, register configuration parameters, and image data after multi-frame overlay. The PL and PS exchange low-speed parameters via the low-speed bus interface GP and image data stream via the high-speed bus interface HP. The algorithm processing modules use the pixel clock as the global clock, and the PL and PS exchange clock uses the AXI bus clock, ensuring controllable timing and deterministic delays. The core functions of each module are as follows: 1) The pose-related coordinate transformation module's core function is to count the number of pixels in a row using a pixel counter `row_cnt` and the number of rows in a frame using a row counter `line_cnt`. The current count values ​​of `row_cnt` and `line_cnt` are used as the read addresses for ROM1 (storing the horizontal pixel offset ratio) and ROM2 (storing the vertical pixel offset ratio), respectively, to read the corresponding offset values. The module then performs coordinate transformation calculations using the offset values ​​`rate_x`, `rate_y`, `1-rate_x`, `1-rate_y`, and the coordinates of the four corner points `xy(0,0)`, `xy(255,0)`, `xy(0,255)`, and `xy(255,255)`. This process is pipelined through two stages of multipliers and two stages of adders, finally outputting the transformed pixel coordinates. The functional diagram is shown below. Figure 4 .

[0038] Figure 4 The ROM bit width and depth for storing horizontal and vertical pixel offset ratios need to be set according to the actual situation. The parameter settings in this module are only exemplary implementations of the present invention and are not limitations on the present invention. Those skilled in the art will understand that, under the premise of conforming to the technical concept of the present invention, they can be adjusted according to the actual image size.

[0039] 2) The area-weighted projection calculation module's core function is to receive the transformed pixel coordinates u2 and v2 output from module 1), perform truncation and rounding up respectively, and obtain interpolated coordinates u3 and u4 from u2 through rounding, and interpolated coordinates v3 and v4 from v2 through rounding. Then, boundary judgment is performed: if u3 and v3 are greater than 0, they are output directly; otherwise, they are set to 0. If u4 is less than the column boundary value line_size and v4 is less than the row boundary value lines, they are output directly; otherwise, they are output as boundary values. The interpolated coordinates and transformed pixel coordinates are calculated and output through an adder and multiplier pipeline to output the area weights k33, k34, k43, and k44 on the four interpolated pixels. The functional diagram is shown below. Figure 5 .

[0040] Figure 5The image boundary parameters mentioned are merely settings for embodiments of the present invention and are not intended to limit the invention. Those skilled in the art will understand that, provided they conform to the technical concept of the present invention, [further details can be found on the other side]. Figure 5 The parameters can also be adjusted or replaced according to the actual situation.

[0041] 3) The image intra-frame fusion module's core function is to align the area weights output by the receiving module 2) with the original image pixels read from the PS DDR in a timely manner, and perform pixel grayscale area allocation calculations through a multiplier; write the two-channel pixel grayscale allocation values ​​to the FIFO for row buffering according to the write enable wr_fifo_en, and then read one channel from the FIFO under the control of the read enable rd_fifo_en, while the other channel is delayed after being read; the other two-channel pixel grayscale allocation values ​​are output directly in one channel and processed by register staking before being output. After aligning the four overlapping pixels, the overlapping pixels are fused through a two-stage adder. The functional diagram is shown below. Figure 6 .

[0042] 4) The multi-frame image start coordinate alignment module's core function is to use shift registers to stitch together the start coordinates (u3, v3) of the four frames of intra-frame fusion output from the receiving module 3). The horizontal coordinate shift register is x_coord{x1,x2,x3,x4}, and the vertical coordinate shift register is y_coord{y1,y2,y3,y4}. A two-stage comparator compares x_coord and y_coord to find the maximum values ​​x_max and y_max of the start coordinates of the four frames, which are then used as the starting coordinates for unifying the four frames. Combined with the address calculation module, the base addresses (bassaddr1, bassaddr2, bassaddr3, and bassaddr4) of the four fused images read from the PS DDR are calculated. The functional diagram is shown below. Figure 7 .

[0043] 5) The multi-channel DDR read / write module's core function is to receive raw image data, pixel clock, and frame synchronization signals acquired by the image sensor, as well as the four-frame fused image data, data validity flag, and the final superimposed image and superimposed image data validity flag output by module 3). This data is then written to the PS DDR in a time-division multiplexing manner via the write control module AXI_WR_DMA_0 connected to the DDR HP0 high-speed data port. The read control module AXI_WR_DMA_0, also connected to the HP0 port, is used to read raw image data and output it to module 3) for intra-frame image fusion. It receives the base address of the four-frame fused image output by module 4) and reads the four-frame image data in parallel via the four-channel DDR read control module connected to the DDR HP1 high-speed data port, achieving alignment and superposition of overlapping image portions. The functional diagram is shown below. Figure 8 .

[0044] Figure 8 The four fused images read from the four-channel DDR controller need to be cropped after being superimposed, so that only the overlapping parts of the images are retained.

[0045] The foregoing has provided a detailed description of an FPGA implementation method and system for multi-frame star map overlay provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A star map attitude correlation parallel overlay system based on FPGA pipelined cache and multi-channel DDR architecture, characterized in that, include: The attitude-related coordinate transformation module is used to receive the coordinates of the four corner points of the current frame image calculated and sent by the PS end based on inertial navigation information, and generate the mapped coordinates of each pixel in the current frame image to the reference coordinate system through pipeline operation according to the relative position of each pixel in the image. An area-weighted projection calculation module is used to receive the mapped coordinates and calculate the area-weighted weight value of each pixel in the current frame image projected onto the adjacent integer pixels in the reference coordinate system based on the overlapping area between the mapped coordinates and the integer pixel coordinates in the reference coordinate system. The image intra-frame fusion module is used to read the original pixel values ​​of the current frame image from DDR, perform grayscale allocation according to the area weighting value, and then perform time-series alignment and superposition of the allocated grayscale values ​​through the line buffer pipeline to generate the intra-frame fused image of the current frame image in the reference coordinate system. The multi-frame image start coordinate alignment module is used to receive the start coordinates of the intra-frame fused images and determine the start coordinates of the overlapping area of ​​all the frames to be superimposed through comparison calculations. The multi-channel DDR read / write control module is used to control the transfer and scheduling of image data between the DDR and various modules, and based on the starting coordinates of the overlapping area, reads the overlapping area data of multiple frames of the intra-frame fused image from the DDR in parallel through multiple channels, and performs the final multi-frame image overlay.

2. The system according to claim 1, characterized in that, The attitude-related coordinate transformation module includes: The pixel counter and the row counter are used to count the horizontal and vertical numbers of the currently processed pixel within the frame, respectively. The first ROM and the second ROM use the horizontal sequence number and the vertical sequence number as read addresses respectively, and store the pre-calculated horizontal pixel offset ratio and vertical pixel offset ratio accordingly; The pipelined multiplication and addition unit is used to perform hierarchical multiplication and addition operations on the horizontal pixel offset ratio, the vertical pixel offset ratio and the coordinates of the four corner points, and output the mapped coordinates of each pixel in a pipelined manner.

3. The system according to claim 2, characterized in that, The area-weighted projection calculation module is specifically used for: The decimal coordinate values ​​of the mapped coordinates are rounded down and up to obtain four adjacent integer pixel coordinates in the reference coordinate system. The validity of the four adjacent integer pixel coordinates is determined, and the out-of-bounds coordinates are clamped. Based on the relative distance between the mapped coordinates and the coordinates of the four adjacent integer pixels, the area weighting value corresponding to the mapped coordinates for the four adjacent integer pixels is calculated by a multiply-accumulate pipeline.

4. The system according to claim 3, characterized in that, The intra-frame image fusion module includes: A timing alignment unit is used to synchronize the area weighting value with the corresponding original pixel value read from DDR. The grayscale allocation operation unit is used to multiply the original pixel value with the corresponding area weighting value to obtain the grayscale components allocated to four adjacent integer pixel positions in the reference coordinate system. A row buffer pipeline unit includes at least one FIFO or row buffer for caching data belonging to different rows in the grayscale component, so as to achieve temporal alignment of overlapping pixels within the same row and between adjacent rows. The pixel overlay unit is used to accumulate all grayscale components at the four adjacent integer pixel positions after alignment to generate the pixel value of the intra-frame fused image.

5. The system according to claim 4, characterized in that, The multi-frame image start coordinate alignment module includes: The shift register group is used to stitch together and store the starting horizontal and starting vertical coordinates of the intra-frame fused images of consecutive input frames; A multi-level comparator is used to compare multiple initial horizontal and initial vertical coordinates after splicing, and select the maximum value as the starting coordinate of the overlapping area; The address calculation unit is used to calculate the read base address of the intra-frame fused image in DDR based on the starting coordinates of the overlapping region and the preset superimposed image size.

6. The system according to claim 5, characterized in that, The multi-channel DDR read / write control module includes: The first DDR control channel is connected to a high-speed data interface of the FPGA, used to write the raw image data acquired by the image sensor into the DDR, and to read the raw image data according to the request of the image intra-frame fusion module. The second DDR control channel is connected to another high-speed data interface of the FPGA and contains multiple parallel read control sub-channels. Each sub-channel is used to read the overlapping region data of an intra-frame fused image in parallel according to the read base address of the corresponding frame calculated by the address calculation unit.

7. The system according to any one of claims 1 to 6, characterized in that, The system uses the FPGA's pixel clock as the working clock for all algorithm processing modules, and the AXI bus clock as the clock for parameter interaction between the PS terminal and the FPGA. The PS terminal and the FPGA transmit corner coordinates and control parameters through a low-speed bus interface and transmit image data through a high-speed data interface.

8. A method for parallel overlay of star map attitude correlation based on FPGA pipelined cache and multi-channel DDR architecture, characterized in that, include: The FPGA receives the coordinates of the four corner points of the current frame image calculated by the PS terminal based on the inertial navigation information and transformed into the reference coordinate system. The FPGA generates the mapped coordinates of each pixel through pipeline operation based on the corner point coordinates and the positional relationship of each pixel in the image. The FPGA calculates the area weighting value of each pixel projected onto adjacent integer pixels based on the overlapping area between the mapped coordinates and the integer pixel coordinates in the reference coordinate system. The FPGA reads the original pixel values ​​of the current frame image from the DDR, allocates grayscale according to the area weighting value, and performs timing alignment and overlay through the line buffer pipeline to generate an intra-frame fused image; The FPGA receives the starting coordinates of the intra-frame fused images from each frame and determines the starting coordinates of the overlapping region of all frames to be superimposed through comparison calculations. The FPGA uses a multi-channel DDR controller to read the overlapping region data of multiple frames of intra-frame fused images in parallel based on the starting coordinates of the overlapping region, and then performs the final superposition.

9. The method according to claim 8, characterized in that, The FPGA reads the original pixel values ​​of the current frame image from the DDR, allocates grayscale values ​​according to the area weighting values, and performs timing alignment and overlay through the line buffer pipeline to generate an intra-frame fused image, including: The area-weighted values ​​are synchronized with the original pixel values ​​read from the DDR in a timing manner; Multiply the synchronized original pixel value by the corresponding weight to obtain the grayscale component; FIFO or row buffering is used to cache different rows of data in the grayscale component to achieve temporal alignment of overlapping pixels within and between rows; All aligned grayscale components are summed to output the pixel values ​​of the intra-frame fused image.

10. The method according to claim 9, characterized in that, The method further includes: The common area image defined by the starting coordinates of the overlapping area and the preset overlay size is cropped from the final overlay image and output as the effective overlay result.

Images