Image processing method, image processing system, fpga-based system, medium and program product

Abstract

The present application relates to an image processing method, an image processing system, an FPGA-based system, a medium and a program product. The method comprises: obtaining, via an arbiter implemented based on programmable logic, coordinates of an original pixel corresponding to a corrected pixel from a coordinate mapping table stored in a DDR memory, the arbiter interacting with a native user interface of the DDR memory via a DDR memory read-write control channel implemented based on programmable logic; obtaining, based on the coordinates of the corresponding original pixel, addresses of the original pixel and a pixel in the same column of the next row of the original pixel in the DDR memory; obtaining, via the arbiter, pixel values of the original pixel, the pixel in the next column of the same row, the pixel in the same column of the next row, and the pixel in the next column of the next row from the DDR memory; obtaining the pixel value of the corrected pixel using bilinear interpolation; and obtaining a corrected image. The present application can directly interact with the DDR native interface based on programmable logic, and realize low-delay, high-flexibility bilinear interpolation image correction.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of image processing, and more specifically to an image processing method, an image processing system, an FPGA-based system, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Traditional image processing methods, particularly those involving bilinear interpolation for image correction, require access to DDR (Double Data Rate Synchronous Dynamic Random Access Memory) via the AXI (Advanced eXtensible Interface) bus ecosystem. However, the typical AXI protocol involves complex address, data, and response handshake logic. During data transfer operations such as pixel reading, the protocol overhead for accessing DDR memory is significant, resulting in high transmission latency.

[0003] In summary, the shortcomings of traditional image encoding methods are: high protocol overhead for accessing DDR memory and high transmission latency. Summary of the Invention

[0004] This invention provides an image processing method, an image processing system, an FPGA-based system, a computer-readable storage medium, and a computer program product. This invention enables direct interaction with the native DDR interface based on programmable logic, achieving low-latency, highly flexible bilinear interpolation image correction.

[0005] According to a first aspect of the present invention, an image processing method is provided. The method includes: obtaining, via an arbitrator, the coordinates of an original pixel in an original image corresponding to a corrected pixel from a coordinate mapping table stored in a DDR memory, wherein the arbitrator interacts with the native user interface of the DDR memory via a DDR memory read / write control channel, and the arbitrator and the DDR memory read / write control channel are implemented based on programmable logic; obtaining, based on the coordinates of the original pixel corresponding to the corrected pixel, the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in the DDR memory; obtaining, via the arbitrator, the pixel values ​​of the original pixel, the pixels in the same column of the next row of the original pixel, and the pixels in the next column of the next row of the original pixel from the DDR memory; obtaining, via bilinear interpolation, the pixel value of the corrected pixel based on the pixel values ​​of the original pixel, the pixels in the same column of the next row of the original pixel, the pixels in the same column of the next row of the original pixel, and the pixels in the next column of the next row of the original pixel; and obtaining a corrected image based on the pixel value of the corrected pixel.

[0006] In some embodiments, the DDR memory read / write control channel includes: a shared channel for writing coordinate mapping table and original image, used to control the burst writing of coordinate mapping table and original image to DDR memory; an original image read control channel, used to control the burst reading of original image from DDR memory; and a coordinate mapping table read control channel, used to read coordinate mapping table from DDR memory.

[0007] In some embodiments, obtaining the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel includes: parsing the coordinates of the original pixel corresponding to the corrected pixel to obtain the integer part of the x-coordinate, the fractional part of the x-coordinate, the integer part of the y-coordinate, and the fractional part of the y-coordinate of the original pixel corresponding to the corrected pixel; and obtaining the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory based on the integer part of the x-coordinate and the integer part of the y-coordinate of the original pixel corresponding to the corrected pixel.

[0008] In some embodiments, the method further includes: in response to determining that the address of the original pixel in the DDR memory is located at the end of a row in the DDR memory, calculating the address of the pixel in the next column of the same row as the original pixel in the DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel; and / or in response to determining that the address of the pixel in the next column of the next row of the original pixel in the DDR memory is located at the end of a row in the DDR memory, calculating the address of the pixel in the next column of the next row of the original pixel in the DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel.

[0009] In some embodiments, the arbitrator is a polling arbitration state machine, which polls the original image writing port, the coordinate mapping table writing port, the coordinate mapping table reading port, and the interpolated pixel reading port.

[0010] In some embodiments, the coordinate mapping table write port has the highest priority.

[0011] In some embodiments, the method further includes: for a corrected pixel, the PL terminal initiates two bursts, wherein one burst is used to obtain the pixel values ​​of the original pixel and the pixel in the next column of the same row of the original pixel from the DDR memory based on the address of the original pixel in the DDR memory, and the other burst is used to obtain the pixel values ​​of the pixel in the next column of the next row of the original pixel and the pixel in the next column of the next row of the original pixel from the DDR memory based on the address of the pixel in the next column of the next row of the original pixel.

[0012] In some embodiments, the method further includes: acquiring 2 to 8 video streams, wherein each video stream includes multiple frames of raw images; and determining whether each video stream undergoes image processing.

[0013] In some embodiments, the method further includes: for multiple frames of original images in a video stream, dividing the addresses of the original images of adjacent frames in DDR memory into a first cache region and a second cache region.

[0014] In some embodiments, the method further includes: for the original images of two adjacent frames, based on a flip signal in the arbitrator, writing one original image of one of the two adjacent frames into a first buffer area at a given time, while reading the other original image from a second buffer area.

[0015] In some embodiments, in response to power-on of the PS terminal, the coordinate mapping table is read from the external memory and sent to the DDR memory.

[0016] According to a second aspect of the present invention, an image processing system is also provided. The image processing system includes: a DDR memory read / write control channel for controlling burst access to the DDR memory; an arbitrator for interacting with the native user interface of the DDR memory via the DDR memory read / write control channel to obtain the coordinates of the original pixel in the original image corresponding to the corrected pixel from a coordinate mapping table stored in the DDR memory; and for obtaining, based on the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in the DDR memory, the pixel in the same column of the next row of the original pixel, and the pixel in the same column of the next row of the original pixel, the image of the original pixel, and the image of the pixel in the next column of the next row of the original pixel from the DDR memory. The system comprises: a pixel value; an address acquisition module, used to obtain the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel; and an interpolation module, used to calculate bilinear interpolation based on the pixel values ​​of the original pixel, the pixels in the same column of the next row of the original pixel, the pixels in the same column of the next row of the original pixel, and the pixels in the next column of the next row of the original pixel, in order to obtain the pixel value of the corrected pixel and obtain the corrected image based on the pixel value of the corrected pixel; wherein, the DDR memory read / write control channel, the arbitrator, the address acquisition module, and the interpolation module are implemented based on programmable logic.

[0017] According to a third aspect of the present invention, an FPGA-based system is also provided, the system including an image processing unit for processing an image using the image processing method according to the first aspect of the present invention.

[0018] According to a fourth aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0019] According to a fifth aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0020] This invention interacts directly with the native user interface of DDR memory through an arbitrator implemented based on programmable logic to obtain the original pixel addresses and pixel values ​​related to bilinear interpolation. This invention enables low-latency, highly flexible bilinear interpolation image correction by directly interacting with the native DDR interface based on programmable logic.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0023] Figure 1 A schematic architecture diagram of an image processing system according to an embodiment of the present invention is shown.

[0024] Figure 2 A flowchart illustrating an image processing method according to some embodiments of the present invention is shown.

[0025] Figure 3 A flowchart illustrating a method for obtaining the addresses of an original pixel and the pixels in the same column of the next row of the original pixel in DDR memory according to some embodiments of the present invention is shown.

[0026] Figure 4 A flowchart illustrating a method for storing video streams according to some embodiments of the present invention is shown.

[0027] Figure 5 A schematic structural diagram of an image processing system according to some embodiments of the present invention is shown.

[0028] Figure 6 A schematic diagram of an image processing flow according to some embodiments of the present invention is shown.

[0029] Figure 7 A schematic diagram of an FPGA-based system according to some embodiments of the present invention is shown.

[0030] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0031] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0032] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0033] As described above, traditional image processing methods for bilinear interpolation image correction tasks require accessing DDR memory via, for example, the AXI bus ecosystem. However, the typical AXI protocol involves complex address, data, and response channel handshake logic. During data transfer operations such as pixel reading, the protocol overhead for accessing DDR memory is significant, resulting in high transmission latency.

[0034] Existing image processing methods, such as traditional FPGA-based image processing methods, typically employ the AXI bus ecosystem, such as VDMA (Video Direct Memory Access) and the AXI SmartConnect interconnect matrix, for data transmission. However, typical AXI protocols involve complex address, data, and response channel handshake logic, with bus protocol overhead sometimes exceeding the data transmission itself, resulting in extremely high data transmission latency and potentially causing pipeline "starvation."

[0035] Furthermore, existing image processing methods, such as those using VDMA-based standard IP, are primarily designed for continuous linear video streams. However, in scenarios where distortion correction is performed using bilinear interpolation, existing image processing methods cannot perform low-level optimization for the non-linear reading of the four corner pixels (the original pixel, the pixel in the same row and next column of the original pixel, the pixel in the next row and same column of the original pixel, and the pixel in the next row and next column of the original pixel) during distortion correction, resulting in a significant waste of DDR memory bandwidth.

[0036] Furthermore, existing image processing methods, such as those based on existing IP like AXI-Stream Switch, struggle to achieve flexible scheduling of distortion correction for some video streams and bypassing of others in multi-stream video scenarios.

[0037] To at least partially address one or more of the aforementioned problems and other potential issues, an exemplary embodiment of the present invention proposes an image processing method. The method includes: obtaining the coordinates of an original pixel in the original image corresponding to the corrected pixel from a coordinate mapping table stored in DDR memory via an arbitrator, wherein the arbitrator interacts with the native user interface of the DDR memory via a DDR memory read / write control channel, and the arbitrator and the DDR memory read / write control channel are implemented based on programmable logic; obtaining the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in the DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel; obtaining the pixel values ​​of the original pixel, the pixel in the next column of the same row of the original pixel, the pixel in the next column of the next row of the original pixel, and the pixel in the next column of the next row of the original pixel from the DDR memory via the arbitrator based on the addresses of the original pixel and the pixel in the next column of the next row of the original pixel; obtaining the pixel value of the corrected pixel using bilinear interpolation based on the pixel values ​​of the original pixel, the pixel in the next column of the same row of the original pixel, the pixel in the next column of the next row of the original pixel, and the pixel in the next column of the next row of the original pixel; and obtaining a corrected image based on the pixel value of the corrected pixel. This invention achieves low-latency, highly flexible bilinear interpolation image correction by directly interacting with the native user interface of the DDR memory through an arbitrator implemented based on programmable logic. The arbitrator uses programmable logic to interact directly with the native DDR interface.

[0038] Figure 1 A schematic architecture diagram of an image processing system 100 according to an embodiment of the present invention is shown. It should be understood that the image processing system 100 may also include additional units not shown and / or the units shown may be omitted, and the scope of the invention is not limited in this respect.

[0039] like Figure 1 As shown, the image processing system includes, for example, a PS (Processing System) terminal 110 and a PL (Programmable Logic) terminal 120. The PL terminal 120 includes at least an arbitrator 121, a DDR memory read / write control channel 124, an address acquisition module 122, and an interpolation module 123. The DDR memory read / write control channel 124, the arbitrator 121, the address acquisition module 122, and the interpolation module 123 are implemented based on programmable logic.

[0040] Among them, PS terminal 110 is connected to PL terminal 120 via a data bus.

[0041] The PS terminal 110 is also interconnected with an external memory 140, for example. The external memory 140 is an SD card, and the PS terminal 110 is connected to the external SD card via an SDIO (Secure Digital Input / Output) controller.

[0042] The arbitrator 121 in the PL terminal 120 interacts with the native user interface of the DDR memory 130 via the DDR memory read / write control channel 124 to enable interconnection between the arbitrator 121 and the DDR memory 130. The address acquisition module 122 and the interpolation module 123 can indirectly access the DDR memory 130 via the arbitrator 121 and the DDR memory read / write control channel 124 to retrieve the required data from the DDR memory 130.

[0043] The following will combine Figure 2 A flowchart describing an image processing method 200 according to an embodiment of the present invention is provided. It should be understood that method 200 can, for example, be executed on an FPGA-based device. Method 200 can also, for example, be executed on a device... Figure 1 The image processing system 100 described Figure 5 The described image processing system 500 or Figure 7 The described FPGA-based system is executed. Method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0044] In step 201, the coordinates of the original pixel in the original image corresponding to the corrected pixel are obtained from the coordinate mapping table stored in the DDR memory via the arbitrator. The arbitrator interacts with the native user interface of the DDR memory via the DDR memory read / write control channel. The arbitrator and the DDR memory read / write control channel are implemented based on programmable logic.

[0045] Regarding the original image, it is, for example, the original frames in the acquired video stream. In some embodiments, the device acquiring the original image is, for example, a wide-angle camera; therefore, the original image may contain distortions such as barrel or pincushion distortions.

[0046] The original image includes, for example, multiple original pixels, such as multiple rows and columns of original pixels. Each original pixel has coordinates, which indicate, for example, the position of the original pixel within the original image. In some embodiments, in a two-dimensional original image, the coordinates of an original pixel are represented, for example, as (x, y), where x indicates the row of the original pixel and y indicates the column of the original pixel; that is, x is the horizontal coordinate and y is the vertical coordinate. In some embodiments, the original image is, for example, in RGB format. x is the horizontal coordinate, representing the column position of the pixel in a row; y is the vertical coordinate, representing the row position of the pixel in a column.

[0047] Regarding the corrected image, the method based on this application can perform image processing on the original image to obtain a corrected image. The corrected image includes, for example, multiple corrected pixels, such as multiple rows and columns of corrected pixels. The corrected image is obtained, for example, based on correction of the original image using bilinear interpolation. Specifically, when correcting using bilinear interpolation, the pixel value of the corrected pixel is determined based on the pixel values ​​of the corresponding original pixel and its neighboring pixels in the original image. For example, the original pixel, the pixel in the same row and next column of the original pixel, the pixel in the next row and next column of the original pixel, and the pixel in the next row and next column of the original pixel. In some embodiments, if the coordinates of the original pixel are (x, y), then the coordinates of the pixel in the same row and next column of the original pixel are (x+1, y), the coordinates of the pixel in the next row and next column of the original pixel are (x, y+1), and the coordinates of the pixel in the next row and next column of the original pixel are (x+1, y+1). In some embodiments, when the original pixel is referred to as P00, the pixel in the same row and next column as the original pixel is referred to as P01, the pixel in the next row and next column as the original pixel is referred to as P10, and the pixel in the next row and next column as the original pixel is referred to as P11. In some embodiments, the above four pixels are referred to as "four corner pixels". In some embodiments, the corrected image is, for example, in RGB format.

[0048] Regarding the coordinate mapping table, it is used, for example, to represent the mapping relationship between the coordinates of corrected pixels and their corresponding original pixels. In some embodiments, since the distortion of the device acquiring the original image (e.g., an image acquisition device, a video acquisition device, etc., such as a camera) is deterministic and can be predetermined, the mapping relationship between the corrected pixels and their corresponding original pixels in the original image based on bilinear interpolation can be deterministic and can be predetermined. The mapping relationship between the coordinates of corrected pixels and their corresponding original pixels can be represented as a coordinate mapping table, which is stored, for example, in DDR memory.

[0049] In some embodiments, in response to power-on of the PS terminal, the coordinate mapping table is read from the external memory and sent to the DDR memory.

[0050] In some embodiments, the operation of reading the coordinate mapping table from the external memory and sending it to the DDR memory is performed, for example, by the PS terminal, the bit splicing buffer module, the arbiter, and the DDR memory read / write control channel. The bit splicing buffer module is described below. Figure 6 The description will not be repeated here. Specifically, the PS end reads the coordinate mapping table from the external memory, sends it to the bit concatenation buffer module on the PL end via the data bus, and further sends it to the arbiter. The arbiter writes the received coordinate mapping table into the DDR memory via the DDR memory read / write control channel. In this way, the coordinates in the coordinate mapping table can be concatenated by the bit concatenation buffer module. After reaching a predetermined number, multiple coordinates are sent to the DDR memory together, improving data transmission efficiency.

[0051] The PS (Processing System) runs, for example, a Linux system. In some embodiments, the PS can pre-calibrate the camera to obtain a coordinate mapping table.

[0052] External storage devices include, for example, external SD cards (Secure Digital Memory Cards) and flash memory. In some embodiments, the external storage device can interact with the PS (Power Switch).

[0053] The methods for reading and sending the coordinate mapping table include, for example, the PS terminal reading the coordinate mapping table generated through calibration from external memory upon power-up. The coordinate mapping table in external memory is also called an offline reverse mapping table, and may be in .bin file format. Furthermore, the coordinate mapping table is sent, for example, to the arbitrator at the PL (Programmable Logic) terminal via a data bus, and the arbitrator writes the coordinate mapping table into DDR memory. In some embodiments, for multiple devices acquiring original images, each has its own corresponding coordinate mapping table, and each corresponding coordinate mapping table is stored in a pre-divided independent Look-Up Table (LUT) storage area in the DDR memory.

[0054] Regarding DDR memory, also known as Double Data Rate Synchronous Dynamic Random Access Memory, it is used, for example, to store data. In some embodiments, DDR memory is used at least to store the original image, that is, at least to store the pixel values ​​of each pixel in the original image. In some embodiments, DDR memory is also used at least to store a coordinate mapping table. In some embodiments, DDR memory is equipped with a corresponding memory interface generator, also known as a MIG (Memory Interface Generator), for direct connection between programmable logic (PL) and DDR memory.

[0055] In some embodiments, the MIG of the DDR memory has a native user interface (NUI) to provide data read and write functionality within the DDR memory. In some embodiments, the native user interface hides the complex command timing details within the DDR, providing an abstraction for data access. The native user interface includes, for example, `app_addr`, `app_cmd`, and `app_wdf_data`. The `app_addr` interface specifies, for example, the address within the DDR memory to be read or written; this address is, for example, a linear address, and is converted via the MIG to a row, column, or bank address within the DDR memory. The `app_cmd` interface specifies, for example, the operation type, including read and write. The `app_wdf_data` interface specifies, for example, the data to be written to the DDR memory. In this way, convenient interaction between the PL and the DDR memory can be achieved without going through the AXI bus, enabling low-latency, highly flexible hardware acceleration.

[0056] Regarding the arbitrator, it is used, for example, to arbitrate based on the state of each port in order to control access to DDR memory. In some embodiments, the arbitrator is implemented based on programmable logic (PL). The arbitrator is, for example, instantiated as hardware circuitry implemented using a pure hardware description language. In some embodiments, each port of the arbitrator is, for example, equipped with a first-in, first-out (FIFO) cache. The arbitrator coordinates the scheduling of read and write requests for each port, for example, based on the water level of each port, where the water level indicates the amount of data stored in the FIFO cache at each port.

[0057] In some embodiments, the arbitrator interacts with the native user interface of the DDR memory via a DDR memory read / write control channel. Specifically, the arbitrator sends data read / write requests to the native user interface of the DDR memory via the DDR memory read / write control channel. The DDR memory read / write control channel controls burst access to the DDR memory, as described below.

[0058] In some embodiments, the arbitrator is a polling arbitration state machine. The polling arbitration state machine includes a raw image writing port, a coordinate mapping table writing port, a coordinate mapping table reading port, and an interpolation pixel reading port, etc., and polls these ports. In this way, it can cycle among these ports, efficiently and fairly processing requests from each port.

[0059] In some embodiments, the coordinate mapping table is written only once when the device is powered on. Therefore, after power-on, the coordinate mapping table writing port does not need to participate in polling. The polling arbitration state machine polls between the original image writing port, the coordinate mapping table reading port, and the interpolated pixel reading port.

[0060] In other embodiments, the coordinate mapping table write port has the highest priority. This ensures that the coordinate mapping table is stored or updated in the DDR memory when image processing is required.

[0061] In some embodiments, the priorities of the arbitrator's ports are fixed. In some embodiments, the interpolation pixel read port has a lower priority than the coordinate mapping table write port, but a higher priority than other ports. This ensures that pixel readout has a higher priority, guaranteeing smooth image processing.

[0062] Regarding the DDR memory read / write control channel, it is used, for example, to control burst access to the DDR memory. In some embodiments, the DDR memory read / write control channel is implemented based on programmable logic (PL). The DDR memory read / write control channel is, for example, instantiated as a hardware circuit implemented based on a pure hardware description language.

[0063] In some embodiments, the DDR memory read / write control channel includes: a shared channel for writing coordinate mapping table and original image, used to control the burst writing of coordinate mapping table and original image to DDR memory; an original image read control channel, used to control the burst reading of original image from DDR memory; and a coordinate mapping table read control channel, used to read coordinate mapping table from DDR memory.

[0064] In some embodiments, a predetermined number of burst accesses to the DDR memory are designated as a group of burst accesses, and the DDR memory read / write control channel performs burst accesses to the DDR memory based on the granularity of the group of burst accesses.

[0065] Regarding the shared channel for writing coordinate mapping tables and original images, it is used, for example, to control burst writes of the coordinate mapping table and original image to DDR memory. The shared channel for writing coordinate mapping tables and original images is, for example, equipped with a cache to cache at least one set of burst write requests. In some embodiments, for the shared channel for writing coordinate mapping tables and original images, 60 burst writes are designated as a set of burst writes. When the cache stores 60 burst writes, i.e., a set of burst writes, a burst write is performed to the DDR memory. During this period, for example, no reads are performed from the DDR memory. In this way, the bandwidth utilization of the DDR memory can be improved.

[0066] Regarding the raw image read control channel, it is used, for example, to control burst reads of the raw image from DDR memory. The raw image read control channel, similar to a coordinate mapping table and a shared channel for raw image writes, is equipped with a buffer to cache at least one set of burst read requests (e.g., 60 reads). In this way, reads corresponding to multiple corrected pixels can be processed simultaneously, and the bandwidth utilization of the DDR memory can be improved.

[0067] Regarding the coordinate map table read control channel, it is used, for example, for burst reads of the coordinate map table from DDR memory. The coordinate map table read control channel, similar to the shared channel for coordinate map table and raw image writes, is also equipped with a buffer to cache at least one set of burst read requests. In some embodiments, one burst corresponds to reading information for 30 512-bit wide coordinates, i.e., obtaining the coordinates of 480 raw pixels. In this way, the coordinates of multiple raw pixels can be obtained at once for address resolution and address translation without frequent DDR memory accesses.

[0068] Regarding the method for obtaining the coordinates of the original pixel corresponding to the corrected pixel, in some embodiments, it includes, for example, burst reading of the coordinate mapping table in the DDR memory via arbitrator control. In a burst, the read bit width is, for example, greater than or equal to the bit width of the corresponding original pixel coordinates. In this case, a selector can be used to select the bit width of the corresponding original pixel coordinates from the read bit width. For example, if the read bit width is, for example, 64, and the bit width of the corresponding original pixel coordinates is, for example, 32, then a routing selector is used to select either the high 32 bits or the low 32 bits of the read bit width for output. It should be understood that the burst reading of the coordinates of the original pixel corresponding to the corrected pixel is, for example, initiated by a corresponding module at the PL end, such as the coordinate parsing module at the PL end, to read the coordinates from the DDR memory.

[0069] In step 202, based on the coordinates of the original pixel corresponding to the corrected pixel, the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in the DDR memory are obtained.

[0070] The coordinates of the original pixel corresponding to the corrected pixel are, for example, represented as (x, y), where x is the horizontal coordinate of the original pixel and y is the vertical coordinate of the original pixel. The coordinates of the pixel in the same column of the next row of the original pixel are, for example, represented as (x, y+1). In some embodiments, x and / or y include an integer part and a fractional part.

[0071] Methods for obtaining the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory include, for example, converting the coordinates of the original pixel into the address of the original pixel in DDR memory; and obtaining the coordinates of the pixel in the same column of the next row of the original pixel based on the coordinates of the original pixel, and converting them into the address of the pixel in the same column of the next row of the original pixel in DDR memory.

[0072] In some embodiments, the coordinates of the original pixel are in (x, y) format; the original image is linearly stored in DDR memory, and its storage address represents a contiguous linear address space, with the address in DDR memory formatted as a single integer. Regarding the method for converting the coordinates of the original pixel to an address in DDR memory, in some embodiments, the address in DDR memory corresponding to the coordinates of the original pixel is calculated based on the base address of the original image, the coordinates of the original pixel, the row stride of the original image, and the unit pixel storage length. The row stride of the original image, for example, indicates the address offset in DDR memory between adjacent rows in the original image, and the address offset includes, for example, valid original pixel data and possible alignment padding. The unit pixel storage length indicates the length of the address range occupied by a single pixel (e.g., including all its components) in memory. Specifically, the pixel value of each pixel includes, for example, multiple components, such as R components, G components, and B components. The unit pixel storage length indicates the specific range covered by all components in the linear address space of the DDR memory.

[0073] For pixels in the same row and column below the original pixel, methods to obtain their coordinates based on the original pixel's coordinates include: if the original pixel's coordinates are (x, y), then the coordinates of the pixel in the same row and column below the original pixel are (x, y+1). Similarly, methods to convert the coordinates of pixels in the same row and column below the original pixel into addresses in DDR memory include calculating the corresponding DDR memory address based on the base address of the original image, the coordinates of the pixel in the same row and column below the original pixel, the row span of the original image, and the address width of each pixel. And so on.

[0074] In some embodiments, the method for obtaining the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in DDR memory includes address resolution and address translation, as described below. Figure 3 The explanation will not be repeated here.

[0075] In some embodiments, the method described above for obtaining the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in the DDR memory is executed by an address acquisition module. The address acquisition module interacts with the arbitrator to access the DDR memory. The address acquisition module is, for example, the address acquisition module 403 described below, and will not be repeated here.

[0076] At step 203, based on the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in the DDR memory, the pixel values ​​of the original pixel, the pixel in the same column of the next row of the original pixel, and the pixel in the next column of the next row of the original pixel are obtained from the DDR memory via the arbitrator.

[0077] In some embodiments, since the addresses of the original pixel and the pixel in the same column of the next row of the original pixel are obtained in the DDR memory, the pixel values ​​of the original pixel and the pixel in the same column of the next row of the original pixel can be read at the corresponding addresses in the DDR memory.

[0078] In some embodiments, the addresses of the original pixel P00 and the pixel P01 in the same row and next column of the original pixel are consecutive in the DDR memory, and the addresses of the pixels P10 and P11 in the next row and next column of the original pixel are consecutive in the DDR memory. Therefore, based on the address of the original pixel P00 in the DDR memory, the address of the pixel P01 in the same row and next column of the original pixel can be obtained; and based on the address of the pixel P10 in the next row and next column of the original pixel in the DDR memory, the address of the pixel P11 in the next row and next column of the original pixel can be obtained.

[0079] Regarding the method of obtaining pixel values ​​from DDR memory, in some embodiments, it is implemented, for example, by initiating a burst read request to the native user interface of the DDR memory. It should be understood that the burst is initiated, for example, by a corresponding module on the PL side. For instance, the burst for reading the original image from the DDR memory may be initiated by the interpolation module.

[0080] In some embodiments, if the original pixel P00 and the pixel P01 in the next column of the same row are located in the same row of the DDR memory, they can be read in the same burst using the bit width of the native user interface. In some embodiments, if the original pixel P10 and the pixel P11 in the next column of the next row are located in the same row of the DDR memory, they can also be read in the same burst using the bit width of the native user interface. In this way, the solution according to this application can obtain the pixel values ​​of the four pixels P00, P01, P10, and P11 required for bilinear interpolation in two bursts, without the need for separate single-point reading of each of the four pixels in conventional methods. In this way, the addressing overhead of the DDR memory is reduced by 50%, the row activation / precharge switching frequency of the DDR memory is reduced by a factor of two, the consumption of on-chip BRAM and logic resources (such as LUTs / FFs) of the FPGA is reduced, the hardware cost is reduced, and furthermore, with limited storage and processing resources, it is possible to handle the processing of more video streams and meet the high-concurrency memory throughput requirements.

[0081] In some embodiments, for a single corrected pixel, the PL terminal initiates two bursts. One burst is used to retrieve the pixel values ​​of the original pixel P00 and the pixel P01 in the next column of the same row from the DDR memory, based on the address of the original pixel P00 in the DDR memory. The other burst is used to retrieve the pixel values ​​of the pixel P10 in the next column of the next row from the DDR memory and the pixel P11 in the next column of the next row from the original pixel. It should be understood that in some embodiments, a single burst can read multiple pixel values ​​located in the same row as P00 or P10 in the DDR memory. For example, based on the address of P00, pixel values ​​including P00, P01, P02, etc., can be read in a single burst. Therefore, for additional pixel values, such as the pixel value of P02, these additional pixel values ​​can be discarded based on a predetermined unit pixel storage length per pixel.

[0082] In some embodiments, if the original pixel P00 and the pixel P01 in the next column of the same row as the original pixel are not located in the same row in the DDR memory, then each of them requires one burst for reading. In other embodiments, if the pixel in the same column of the next row of the original pixel and the pixel in the next column of the next row of the original pixel are not located in the same row in the DDR memory, then each of them requires one burst for reading. That is to say, the pixel values ​​of the four pixels P00, P01, P10, and P11 required for bilinear interpolation require two to four bursts.

[0083] It should be understood that, regarding the processing of whether adjacent pixels in the same row of the original image are located in the same row in the DDR memory, in some embodiments, it can be determined in advance whether the original pixel P00 and the pixel P01 in the next column of the same row of the original pixel are located in the same row in the DDR memory, and / or whether the pixel P10 in the next column of the next row of the original pixel and the pixel P11 in the next column of the next row of the original pixel are located in the same row in the DDR memory, and accordingly determine whether it is necessary to calculate the addresses of the pixel P01 in the next column of the same row of the original pixel and / or the pixel P11 in the next column of the next row of the original pixel in the DDR memory, so as to perform burst read.

[0084] In step 204, the pixel value of the corrected pixel is obtained by bilinear interpolation based on the pixel values ​​of the original pixel, the pixel in the same row and next column of the original pixel, the pixel in the next row and next column of the original pixel.

[0085] Bilinear interpolation is a typical method used in image processing. Bilinear interpolation involves performing two linear interpolations in the row direction and one linear interpolation in the column direction based on the pixel values ​​of the original pixel, the pixels in the same row and next column of the original pixel, the pixels in the next row and next column of the original pixel, and the pixels in the next row and next column of the original pixel, to obtain the pixel value of the corrected pixel.

[0086] In some embodiments, bilinear interpolation can be implemented based on programmable logic (PL). In some embodiments, the method for obtaining the pixel value of the corrected pixel is implemented, for example, by the interpolation module 404 described below.

[0087] In step 205, a corrected image is obtained based on the pixel value of the corrected pixel.

[0088] One method for obtaining a corrected image includes, for example, calculating the pixel value of each corrected pixel in the corrected image and stitching them together to obtain the corrected image.

[0089] In this manner, according to the scheme of this application, the coordinates of the original pixel in the original image corresponding to the corrected pixel are obtained from the coordinate mapping table stored in the DDR memory via an arbitrator implemented based on programmable logic. The arbitrator interacts with the native user interface of the DDR memory via the DDR memory read / write control channel. The arbitrator and the DDR memory read / write control channel are implemented based on programmable logic. Based on the coordinates of the original pixel corresponding to the corrected pixel, the addresses of the original pixel and the pixel in the same column of the next row in the DDR memory are obtained. The pixel values ​​of the original pixel, the pixel in the next column of the same row, the pixel in the next row of the same column, and the pixel in the next row of the next column are obtained from the DDR memory via the arbitrator. The pixel value of the corrected pixel is obtained using bilinear interpolation. The corrected image is then obtained. This invention can directly interact with the native DDR interface based on programmable logic, without instantiating a large VDMA or SmartConnect IP matrix, reducing the redundant overhead of the handshake cycle brought by the AXI bus, and achieving low-latency, highly flexible bilinear interpolation image correction.

[0090] The following will combine Figure 3 This invention describes a method 300 for obtaining the addresses of an original pixel and the pixels in the same column of the next row of the original pixel in DDR memory. It should be understood that method 300 can be performed, for example, at an FPGA-based device. Method 300 can also be performed, for example, at... Figure 1 The image processing system 100 described Figure 5 The described image processing system 500 or Figure 7 The described FPGA-based system is executed. Method 300 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0091] At step 302, the coordinates of the original pixel corresponding to the corrected pixel are parsed to obtain the integer part of the x-coordinate, the fractional part of the x-coordinate, the integer part of the y-coordinate, and the fractional part of the y-coordinate of the original pixel corresponding to the corrected pixel.

[0092] In some embodiments, the method for resolving the coordinates of the original pixel corresponding to the corrected pixel includes, for example, truncating the coordinates of the original pixel corresponding to the corrected pixel to obtain the resolved coordinates of the corresponding original pixel. These coordinates include, for example, the integer part of the x-coordinate, the fractional part of the x-coordinate, the integer part of the y-coordinate, and the fractional part of the y-coordinate. For instance, the bit width of the corresponding original pixel coordinates is 32 bits, where the first 16 bits are the x-coordinate and the last 16 bits are the y-coordinate. In the first 16 bits, the highest 11 bits are the integer part of the x-coordinate, and the second highest 5 bits are the fractional part of the x-coordinate; similarly, in the last 16 bits, the highest 11 bits are the integer part of the y-coordinate, and the second highest 5 bits are the fractional part of the y-coordinate. By truncating the coordinates of the original pixel corresponding to the corrected pixel, the corresponding integer and fractional parts can be obtained.

[0093] At step 304, based on the integer parts of the horizontal and vertical coordinates of the original pixel corresponding to the corrected pixel, the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in the DDR memory are obtained.

[0094] Regarding methods for obtaining the addresses of the original pixel and the pixels in the same column of the next row in DDR memory, in some embodiments, the address of the original pixel in DDR memory is calculated based on the base address of the original image, the integer parts of the x-coordinate and y-coordinate of the original pixel, the row span of the original image, and the address width of each pixel. The row span of the original image, for example, indicates the address width of a row of pixels in the original image in DDR memory. In some embodiments, the calculation relationship is represented as: Address of the R component of the original pixel in DDR memory = Base address of the original image + (Integer part of the y-coordinate) (Row span of the original image) + (Integer part of the x-axis) (Unit pixel storage length). The address of the G component in DDR memory = the address of the R component in DDR memory + 1, and the address of the B component in DDR memory = the address of the R component in DDR memory + 2. The unit pixel storage length (or pixel span) is, for example, 4 bytes, corresponding to 4 consecutive byte address units in DDR memory. The above addresses are in bytes.

[0095] It should be understood that the ordinate of a pixel in the same column and row below the original pixel is equal to the ordinate of the original pixel plus 1, and its x-coordinate is equal to the x-coordinate of the original pixel. Similarly, the relationship between the x and ordinate of the original pixel, the pixels in the same column and row below the original pixel, the pixels in the same column and row below the original pixel, and the pixels in the next column and row below the original pixel can be understood.

[0096] In some embodiments, it is determined whether the address of the original pixel and / or the pixel in the same column of the next row in the DDR memory is located at the end of a row in the DDR memory. If the address of the original pixel in the DDR memory is located at the end of a row in the DDR memory, and the pixel in the same column of the next row of the original pixel is located in another row of the DDR memory, then their addresses are calculated separately for burst reads. Specifically, in response to determining that the address of the original pixel in the DDR memory is located at the end of a row in the DDR memory, the address of the pixel in the same column of the next row of the original pixel is calculated based on the coordinates of the original pixel corresponding to the corrected pixel. For example, in response to determining that the address of the original pixel in the DDR memory is located at the end of a row in the DDR memory, the integer parts of the x-coordinate and y-coordinate of the pixel in the same column of the next row of the original pixel are calculated based on the integer parts of the x-coordinate and y-coordinate of the original pixel. Further, the address of the pixel in the DDR memory is calculated based on the integer parts of the x-coordinate and y-coordinate of the pixel in the same column of the next row of the original pixel.

[0097] Similarly, in some embodiments, in response to determining that the address of the pixel in the same column of the next row of the original pixel is located at the end of the row in the DDR memory, the address of the pixel in the next column of the next row of the original pixel is calculated in the DDR memory based on the coordinates of the original pixel.

[0098] In this way, for two corner pixels (e.g., P00 and P01; P10 and P11) in the same row of the original image but located in different rows in DDR memory, their addresses can be calculated separately and burst reads can be performed separately.

[0099] It should be understood that the image processing method described above can be used to process images from multiple video streams simultaneously. In some embodiments, the method described above further includes: acquiring 2 to 8 video streams, wherein each video stream includes multiple frames of original images; and determining whether each video stream requires image processing.

[0100] Regarding video streams, these are acquired, for example, by video acquisition devices. A video stream comprises multiple frames of raw images. Regarding multiple video streams, these are acquired, for example, by multiple video acquisition devices. In some embodiments, the multiple video streams are 2 to 8 video streams. Each video stream's corresponding video acquisition device may have its own distortion characteristics, which may be the same or different. In some embodiments, at least one of the multiple video streams is not subjected to distortion correction, i.e., the image processing described above is not performed.

[0101] Regarding methods for determining whether each video stream undergoes image processing, in some embodiments, the image processing described above is not performed on a certain video stream based on a pre-set condition. In some embodiments, the determination of whether each video stream undergoes image processing is based on the AXI-Lite signal in real time. Regarding methods for not performing the image processing described above on a certain video stream, for example, by... Figure 4 The bypass scheduling module 405 bypasses this video stream.

[0102] The following will combine Figure 4 A video stream storage method 400 according to an embodiment of the present invention is described. In this method, for at least two adjacent frames of the original images in the video stream, a corresponding corrected image is obtained by performing the image processing method described above based on the original images in the video stream. It should be understood that method 400 can be executed, for example, at a device based on an FPGA chip. Method 400 can also be executed, for example, at... Figure 1 The image processing system 100 described Figure 5 The described image processing system 500 or Figure 7 The described FPGA-based system is executed. Method 400 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0103] At step 402, for multiple frames of original images in the video stream, the addresses of the original images of adjacent frames in the DDR memory are divided into a first cache region and a second cache region.

[0104] The first and second buffer regions are, for example, two statically partitioned buffer regions in DDR memory, used to buffer consecutive frames within adjacent frames of a video stream. In some embodiments, the partitioning of the first and second buffer regions is predetermined. In some embodiments, for multiple video streams, each video stream has two corresponding statically partitioned buffer regions in DDR memory.

[0105] At step 404, for the original images of two adjacent frames, based on the flip signal in the arbitrator, one original image of the two adjacent frames is written to the first buffer area at a given time, while the other original image is read from the second buffer area.

[0106] Regarding the toggle signal, it is, for example, a level signal. The toggle signal, for example, indicates the operation type of each buffer region, that is, whether each buffer region is read or written. In some embodiments, a high-level signal indicates that the first buffer region is read and the second buffer region is written, and a low-level signal indicates that the first buffer region is written and the second buffer region is read. Or conversely, a low-level signal indicates that the first buffer region is read and the second buffer region is written, and a high-level signal indicates that the first buffer region is written and the second buffer region is read.

[0107] Regarding the operation of each buffer region, in some embodiments, at a given time, one frame of the original image from two adjacent frames is written to one buffer region, while another frame of the original image is read from the other buffer region. For example, at time T, the two adjacent frames are the Nth frame and the (N-1)th frame. The (N-1)th frame is read from the second buffer region, for example, for image processing of the (N-1)th frame; and simultaneously, the Nth frame is written to the first buffer region. Next, at time T+1, the signal is toggled, the Nth frame is read from the first buffer region and used for image processing, for example; simultaneously, the next frame, the (N+1)th frame, is written over the second buffer region, and so on. In this way, the first and second buffer regions can be read or written alternately. And at a given time, one of the first and second buffer regions is read while the other is written, thereby ensuring that the preceding and following frames in the video stream are processed sequentially, avoiding image tearing.

[0108] It should be understood that in some embodiments, the original image read at step 404 is, for example, the original image corresponding to the original pixels obtained in step 203 of the method 200 described above.

[0109] In this way, the first and second buffer areas can implement a dual-buffering mechanism or a "ping-pong buffering" mechanism, which allows the processing of adjacent frames in the video stream to be independent and avoids confusion.

[0110] The following will combine Figure 5 An image processing system 500 according to an embodiment of the present invention is described. It should be understood that system 500 is used, for example, to perform image processing according to the image processing methods 200, 300, and 400 described above.

[0111] In some embodiments, the image processing system 500 includes an arbitrator 502, an address acquisition module 503, an interpolation module 504, and a DDR memory read / write control channel 506.

[0112] In some embodiments, the image processing system 500 further includes a raw image acquisition module 501, which is used, for example, to acquire a raw image, the raw image comprising a plurality of raw pixels. Regarding the raw image, refer to the above description, and will not be repeated here.

[0113] Regarding the arbitrator 502, the arbitrator 502 is implemented based on programmable logic and is used to interact with the native user interface of the DDR memory via the DDR memory read / write control channel 506 in order to obtain the coordinates of the original pixel corresponding to the corrected pixel from the coordinate mapping table stored in the DDR memory; and to obtain the pixel values ​​of the original pixel, the pixel in the same row and next column of the original pixel, the pixel in the same row and next column of the original pixel, and the pixel values ​​of the pixel in the next row and next column of the original pixel from the DDR memory based on the addresses of the original pixel and the pixel in the same column of the next row of the original pixel.

[0114] In some embodiments, the arbitrator 502 is a polling arbitration state machine. In the polling arbitration state machine, the original image acquisition module 501 and the coordinate mapping table share a channel with the original image writing port; the bit splicing buffer module and the coordinate mapping table share a channel with the original image writing port; the address acquisition module 503 and the coordinate mapping table reading control channel are connected to the coordinate mapping table reading port; and the interpolation module 504 and the original image reading control channel are connected to the interpolation pixel reading port. It should be understood that the arbitrator 502 and its functions can be further explained by referring to the content described above, and will not be repeated here.

[0115] In some embodiments, the arbitrator 502 interacts with the native user interface of the external DDR memory 510 via the DDR memory read / write control channel 506.

[0116] Regarding the address acquisition module 503, it is implemented based on programmable logic. For example, it is used to obtain the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel. The method for obtaining the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in DDR memory is described above and will not be repeated here.

[0117] Regarding the interpolation module 504, it is implemented based on programmable logic. For example, it is used to calculate bilinear interpolation based on the pixel values ​​of the original pixel, the pixel in the same row and next column of the original pixel, the pixel in the next row and next column of the original pixel, and the pixel values ​​of the next row and next column of the original pixel, in order to obtain the pixel value of the corrected pixel and obtain the corrected image based on the pixel value of the corrected pixel. The method for calculating bilinear interpolation to obtain the pixel value of the corrected pixel is described above and will not be repeated here.

[0118] Regarding the DDR memory read / write control channel 506, it is implemented based on programmable logic, for example, for controlling burst access to the DDR memory. Referring to the context description, further details are omitted here.

[0119] In some embodiments, the address acquisition module 503 further includes two sub-modules: a coordinate parsing module 532 and an address conversion module 534.

[0120] Regarding the coordinate parsing module 532, it is used, for example, to parse the coordinates of the original pixel corresponding to the corrected pixel, so as to obtain the integer part of the x-coordinate, the fractional part of the x-coordinate, the integer part of the y-coordinate, and the fractional part of the y-coordinate of the original pixel corresponding to the corrected pixel. In some embodiments, the coordinate parsing module 532 performs step 302 as described above, for example, and will not be repeated here.

[0121] Regarding the address translation module 534, it is used, for example, to obtain the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory based on the integer parts of the horizontal and vertical coordinates of the original pixel corresponding to the corrected pixel. The address translation module 434 performs step 304 as described above, for example, and will not be repeated here.

[0122] In some embodiments, the image processing system 500 further includes a bypass scheduling module 505, which is used, for example, to acquire 2 to 8 video streams, wherein each video stream includes multiple frames of original images; and to determine whether each video stream is to be processed.

[0123] In some embodiments, the image processing system 500 further includes a bit splicing buffer module for splicing coordinates in a coordinate mapping table.

[0124] Regarding the interaction between the various modules in the Image Processing System 500, the following is a combination of... Figure 6The image processing flow is described below. In some embodiments, camera calibration is performed at the PS terminal 601 to generate a coordinate mapping table and store it to an external SD card 602. The coordinate mapping table is, for example, in .bin format. In some embodiments, when the PS terminal 601 is powered on, the coordinate mapping table is sent to the PL terminal via the data bus. Specifically, in response to the PS terminal 601 being powered on, the PS terminal reads the coordinates from the coordinate mapping table from the external SD card 602, performs multiple coordinate splicing via the bit splicing buffer module 609 and sends it to the arbitrator 603, and writes it to the DDR memory 604 via the coordinate mapping table and the original image writing shared channel 621. In some embodiments, regarding the transmission process of the coordinate mapping table, the coordinate mapping table is transmitted from the external SD card 602 to the PS terminal 601. The PS terminal 601 is connected to the bit splicing buffer module 609 of the PL terminal. The bit splicing buffer module 609 splices multiple coordinate data, for example, splicing 32-bit coordinate data into 512-bit coordinate data, so as to send them in parallel to the arbitrator 603, and writes them to the DDR memory 604 via the coordinate mapping table and the original image writing common channel 621.

[0125] In some embodiments, multiple video streams are obtained via multiple video acquisition devices. For example, a bypass scheduling module 605 first determines whether each video stream requires image processing. In some embodiments, the bypass scheduling module 605 receives, for example, an AXI-Lite signal indicating whether each video stream requires image processing, and sends this signal to the corresponding module. In some embodiments, the bypass scheduling module 605 is also referred to as a "routing arbitrator." This allows for flexible independent arbitration of each video stream within the same system. If a video stream does not require image processing, it is directly passed through the bypass scheduling module 605 to the next-level module 607, skipping image processing and avoiding unnecessary consumption of transmission resources.

[0126] In some embodiments, the unprocessed video stream is sent to a lower-level module 607, which may be used, for example, to perform further image processing or image output. In some embodiments, the lower-level module 607 may be used, for example, to output the received image or video to a display screen, or for example, to perform image enhancement on the received image or video.

[0127] In some embodiments, the video stream to be processed is sent to a video stream converter 606, which, for example, converts different video stream formats, such as video streams with custom network protocols or AXI-Stream streams, into data-valid signals and image data. In some embodiments, the video stream converter 606 is, for example, part of the raw image acquisition module 501 described above. In some embodiments, the video stream converter 606 is connected to the raw image write port of the arbitrator 603, thereby writing each raw image in the video stream converted by the video stream converter 606 to the DDR memory 604 via the arbitrator 603.

[0128] In some embodiments, the coordinate parsing module 611 is connected to the coordinate mapping table reading port of the arbitrator 603. For a corrected pixel, the coordinate parsing module 611 obtains the coordinates of the original pixel in the original image corresponding to the corrected pixel from the coordinate mapping table stored in the DDR memory 604 via the arbitrator 603 and the coordinate mapping table reading control channel 623.

[0129] In some embodiments, the coordinate parsing module 611 parses the coordinates of the original pixels in the original image corresponding to the corrected pixels, and sends the parsed coordinates to the address conversion module 612 for coordinate-to-address conversion. The coordinate parsing module 611 and the address conversion module 612 transmit data, for example, based on a handshake signal. The coordinate parsing module 611, for example, refers to... Figure 5 The coordinate analysis module 532 will not be described in detail here.

[0130] In some embodiments, the address conversion module 612 obtains the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in the DDR memory based on the integer parts of the horizontal and vertical coordinates of the original pixel corresponding to the corrected pixel. The method for converting coordinates to addresses is described above and will not be repeated here.

[0131] In some embodiments, the address translation module 612, based on the calculated address, determines whether the original pixel P00 and the pixel P01 in the next column of the same row as the original pixel are located in the same row in the DDR memory, and / or whether the pixel P10 in the next column of the next row as the original pixel and the pixel P11 in the next column of the next row as the original pixel are located in the same row in the DDR memory. It then determines whether it is necessary to calculate the addresses of the pixel P01 in the next column of the same row as the original pixel and / or the pixel P11 in the next column of the next row as the original pixel in the DDR memory, and determines the required number of bursts. The address translation module 612, for example, refers to... Figure 5 The address translation module 534 will not be described in detail here.

[0132] In some embodiments, based on the addresses of the original pixel and the pixels in the same column of the next row of the original pixel in the DDR memory obtained by the address translation module 612, the arbitrator 603 interacts with the native user interface of the DDR memory 604 via the original image read control channel 622 to obtain the pixel values ​​of the original pixel, the pixels in the same column of the next row of the original pixel, the pixels in the same column of the next row of the original pixel, and the pixels in the next column of the next row of the original pixel. The pixel values ​​of the above four corner pixels are sent to the interpolation module via the original image read control channel 622 and the arbitrator 603. The interpolation module uses bilinear interpolation to calculate the pixel value of the corrected pixel. Figure 6 In the illustrated embodiment, the address translation module 612 and the interpolation module are implemented as the same module, for example. In some embodiments, the pixel values ​​of the four corner pixels are obtained, for example, in two, three, or four bursts. The interpolation module waits for the pixel values ​​of the four corner pixels to be aligned, that is, after the pixel values ​​of the four corner pixels are all obtained, before performing bilinear interpolation, thus realizing the "read cache aligned interpolation" function.

[0133] In some embodiments, after obtaining the pixel values ​​of each corrected pixel in the corrected image, the corrected image is obtained by stitching the pixels together.

[0134] In some embodiments, one or more of the bit splicing buffer module 609, video stream converter 606, coordinate parsing module 611, address conversion module 612, and interpolation module are configured, for example, with an asynchronous FIFO (such as...). Figure 6 (As shown in "FIFO"), this allows for cross-clock domain processing. In some embodiments, the arbiter 603, the coordinate map and original image write shared channel 621, the original image read control channel 622, and the coordinate map read control channel 623 operate, for example, in a 300MHz clock domain for output to the user. In some embodiments, the unit pixel storage length of the original pixel is, for example, 4 bytes, i.e., 24 bits wide, and the bit width of the original pixel coordinates is, for example, 32 bits. Both need to be concatenated into 512-bit wide data for simultaneous writing to DDR memory. In some embodiments, the coordinate resolution module 611 and the interpolation module 612 need to operate in a 200MHz clock domain; reducing the clock domain can, for example, improve timing stability. In this way, the asynchronous FIFO enables the system to process across clock domains.

[0135] In some embodiments, the corrected image or the corresponding video stream is converted to a predetermined specific format via the specific format conversion module 608 and sent to the lower-level module 607 for further image processing or image output. In some embodiments, the specific format conversion module 608 converts RGB format to an AXI-Stream or a video stream based on a custom network protocol. In this way, lightweight video protocols can be flexibly adapted.

[0136] In this way, the solution according to this application can utilize the native user interface of DDR memory to achieve high-efficiency, high-concurrency distortion correction of multiple video streams.

[0137] The following will combine Figure 7 An FPGA-based system 700 is described, illustrating an image processing method according to an embodiment of the present invention. It should be understood that the FPGA-based system 700 includes at least a PL terminal, and may also include a PS terminal and / or DDR memory. It should also be understood that the FPGA-based system 700 may further include additional units not shown and / or the units shown may be omitted; the scope of the invention is not limited in this respect.

[0138] like Figure 7 As shown, the FPGA-based system 700 includes at least an image processing unit 702 for processing images according to the image processing methods 200, 300, or 400 described above. It should be understood that the FPGA-based system 700 can be integrated into other computing systems.

[0139] This invention can be a method, an FPGA-based system, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0140] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0141] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0143] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0144] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors.

Claims

1. An image processing method, characterized in that, include: The coordinates of the original pixels in the original image corresponding to the corrected pixels are obtained from the coordinate mapping table stored in the DDR memory via the arbiter. The arbiter interacts with the native user interface of the DDR memory via the DDR memory read / write control channel. The arbiter and the DDR memory read / write control channel are implemented based on programmable logic. Based on the coordinates of the original pixel corresponding to the corrected pixel, the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory are obtained. Based on the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in the DDR memory, the pixel values ​​of the original pixel, the pixel in the same column of the next row of the original pixel, the pixel in the same column of the next row of the original pixel, and the pixel in the next column of the next row of the original pixel are obtained from the DDR memory via the arbitrator. The pixel value of the corrected pixel is obtained using bilinear interpolation, based on the pixel values ​​of the original pixel, the pixels in the same row and next column of the original pixel, the pixels in the next row and next column of the original pixel, and the pixels in the next row and next column of the original pixel; and The corrected image is obtained based on the pixel values ​​of the corrected pixels.

2. The method according to claim 1, characterized in that, DDR memory read / write control channels include: The coordinate mapping table and the original image are written to a shared channel, which is used to control the burst writing of the coordinate mapping table and the original image to the DDR memory; The raw image read control channel is used to control the burst read of raw images from DDR memory; and The coordinate mapping table read control channel is used for burst reads of the coordinate mapping table from DDR memory.

3. The method according to claim 1, characterized in that, Based on the coordinates of the original pixel corresponding to the corrected pixel, the addresses of the original pixel and the pixels in the same column of the next row in the DDR memory are obtained, including: Parse the coordinates of the original pixel corresponding to the corrected pixel to obtain the integer part, fractional part, integer part, and fractional part of the x-coordinate of the original pixel corresponding to the corrected pixel; and Based on the integer parts of the x-coordinate and y-coordinate of the original pixel corresponding to the corrected pixel, the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory are obtained.

4. The method according to claim 3, characterized in that, The method also includes: In response to determining that the address of the original pixel in DDR memory is located at the end of a row in DDR memory, the addresses of the pixels in the next column of the same row as the original pixel are calculated in DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel; and / or In response to determining that the address of the pixel in the same column of the next row of the original pixel is located at the end of the row in the DDR memory, the address of the pixel in the next column of the next row of the original pixel is calculated in the DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel.

5. The method according to claim 1, characterized in that, The arbitrator is a polling arbitration state machine, which polls the original image writing port, the coordinate mapping table writing port, the coordinate mapping table reading port, and the interpolated pixel reading port.

6. The method according to claim 5, characterized in that, The coordinate mapping table is written to the port with the highest priority.

7. The method according to claim 1, characterized in that, The method also includes: For a single corrected pixel, the PL terminal initiates two bursts. One burst is used to retrieve the pixel values ​​of the original pixel and the next pixel in the same row and column from the DDR memory based on the address of the original pixel in the DDR memory. The other burst is used to retrieve the pixel values ​​of the next pixel in the next row and column from the DDR memory based on the address of the next pixel in the next row and column from the DDR memory.

8. The method according to claim 1, characterized in that, The method also includes: Acquire 2 to 8 video streams, where each video stream includes multiple frames of raw images; and Determine whether image processing is required for each video stream.

9. The method according to claim 1, characterized in that, The method also includes: For multiple frames of original images in a video stream, the addresses of the original images of adjacent frames in DDR memory are divided into the first cache region and the second cache region.

10. The method according to claim 9, characterized in that, The method also includes: For two adjacent frames of original images, based on the flip signal in the arbitrator, one frame of the original image is written to the first buffer area at a given time, while the other frame of the original image is read from the second buffer area.

11. The method according to claim 1, characterized in that, In response to power-on at the PS end, the coordinate mapping table is read from the external memory and sent to the DDR memory.

12. An image processing system, characterized in that, include: DDR memory read / write control channel, used to control burst access to DDR memory; The arbitrator interacts with the native user interface of the DDR memory via the DDR memory read / write control channel to obtain the coordinates of the original pixel in the original image corresponding to the corrected pixel from the coordinate mapping table stored in the DDR memory; and obtains the pixel values ​​of the original pixel, the pixel in the same column of the same row as the original pixel, the pixel in the same column of the next row as the original pixel, and the pixel value of the pixel in the next column of the next row as the original pixel from the DDR memory based on the addresses of the original pixel and the pixel in the same column of the next row as the original pixel in the DDR memory. The address acquisition module is used to obtain the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory based on the coordinates of the original pixel corresponding to the corrected pixel. as well as The interpolation module is used to calculate bilinear interpolation based on the pixel values ​​of the original pixel, the pixel in the same row and next column of the original pixel, the pixel in the next row and next column of the original pixel, and the pixel values ​​of the next row and next column of the original pixel, so as to obtain the pixel value of the corrected pixel and obtain the corrected image based on the pixel value of the corrected pixel. The DDR memory read / write control channel, arbitrator, address acquisition module, and interpolation module are implemented based on programmable logic.

13. The image processing system according to claim 12, characterized in that, DDR memory read / write control channels include: The coordinate mapping table and the original image are written to a shared channel, which is used to control the burst writing of the coordinate mapping table and the original image to the DDR memory; The raw image read control channel is used to control the burst read of raw images from DDR memory; and The coordinate mapping table read control channel is used for burst reads of the coordinate mapping table from DDR memory.

14. The image processing system according to claim 12, characterized in that, The address acquisition module includes: The coordinate parsing module is used to parse the coordinates of the original pixel corresponding to the corrected pixel, in order to obtain the integer part of the x-coordinate, the decimal part of the x-coordinate, the integer part of the y-coordinate, and the decimal part of the y-coordinate of the original pixel corresponding to the corrected pixel; and The address translation module is used to obtain the addresses of the original pixel and the pixel in the same column of the next row of the original pixel in DDR memory based on the integer parts of the horizontal and vertical coordinates of the original pixel corresponding to the corrected pixel.

15. The image processing system according to claim 13, characterized in that, The image processing system further includes: a raw image acquisition module for acquiring a raw image, the raw image including multiple raw pixels; and a bit stitching cache module for stitching coordinates in a coordinate mapping table; The arbitrator is a polling arbitration state machine, which includes: an original image writing port, a coordinate mapping table writing port, a coordinate mapping table reading port, and an interpolation pixel reading port. The original image acquisition module, the coordinate mapping table, and the original image writing share a channel connected to the original image writing port. The bit splicing buffer module, the coordinate mapping table, and the original image writing share a channel connected to the coordinate mapping table writing port. The address acquisition module and the coordinate mapping table reading control channel are connected to the coordinate mapping table reading port. The interpolation module and the original image reading control channel are connected to the interpolation pixel reading port.

16. The image processing system according to claim 12, characterized in that, The system also includes: The bypass scheduling module is used to acquire 2 to 8 video streams, where each video stream includes multiple frames of raw images; and to determine whether each video stream needs to undergo image processing.

17. An FPGA-based system, characterized in that, include: An image processing unit is configured to process an image according to the image processing method as described in any one of claims 1-11.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method according to any one of claims 1-11.

19. A computer program product, characterized in that, Includes a computer program, which, when executed by a machine, performs the method according to any one of claims 1-11.

Images