An in-memory index-based image super-resolution reconstruction hardware acceleration method and system

Abstract

The application provides an in-memory index-based image super-resolution reconstruction hardware acceleration method and system, which replaces a one-word pixel block index with a cross-shaped pixel block, and replaces the rotation of the whole image with an r*r high-resolution pixel block arrangement designed according to four directions of 0°, 90°, 180° and 270°, thereby simplifying the rotation operation of image adaptive enhancement which is difficult to realize in hardware, greatly saving the read-write data amount, and facilitating more sufficient and reasonable allocation of hardware resources. Furthermore, the application reasonably designs a parallelization acceleration function according to the storage resources of the FPGA and the size of the lookup table (LUT), divides the picture into multiple blocks, reads and calculates the LUT for each block of data, realizes multi-LUT parallelization reading and calculation, arranges the results according to the segmentation to finally obtain an HR image, and greatly reduces the processing speed.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, specifically relating to a hardware acceleration method and system for image super-resolution reconstruction based on in-memory indexing. Background Technology

[0002] In recent years, with the development of artificial intelligence, computer vision algorithms, and image sensing systems, high spatial resolution images are frequently desired in electronic imaging applications such as medical care, military security, remote sensing, and video games. High spatial resolution means that the acquired images can provide more detailed information to aid in accurate scene perception. The most direct way to obtain high-resolution images is to use high-resolution image sensors, but due to limitations in manufacturing processes and costs, high-resolution image sensors are difficult to deploy on a large scale. Therefore, utilizing existing low-resolution imaging systems to acquire high-resolution (HR) images through super-resolution technology has significant practical value.

[0003] Traditional super-resolution reconstruction methods can be categorized into several types, including interpolation-based, reconstruction-based, and sparse coding-based methods. Data-driven deep learning methods offer better scene adaptability compared to traditional super-resolution reconstruction methods, and therefore have seen significant research and application in recent years. Existing classic deep learning algorithms for image super-resolution include: Super-Resolution Convolutional Neural Network (SRCNN), Fast Super-Resolution Convolutional Neural Network (FSRCNN), Deep Convolutional Network (VDSR), Deep Recurrent Convolutional Network (DRCN), Subpixel Convolutional Neural Network (ESPCN), and Generative Adversarial Network (SRGAN).

[0004] While deep learning-based super-resolution algorithms have demonstrated significant performance advantages, they still suffer from high computational and storage requirements, as well as insufficient inference speed, making them difficult to deploy on mobile devices with limited hardware resources and power consumption. To address this issue, the industry has proposed an efficient lookup table method for image super-resolution (SR-LUT). However, its adaptive resolution enhancement operation, which expands the receptive field to ensure accuracy, still makes it difficult to implement on resource-constrained edge hardware. Furthermore, this method lacks efficient acceleration strategies tailored to the structural characteristics of the LUT. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a hardware acceleration method and system for image super-resolution reconstruction based on in-memory indexing. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a hardware acceleration method for image super-resolution reconstruction based on in-memory indexing, applied on an FPGA, the hardware acceleration method for image super-resolution reconstruction based on in-memory indexing comprising:

[0007] S100: Read the pixel values ​​of the original image and the lookup table from the SD card in three-line units, and place the pixel values ​​of the original image into the first cache space and the lookup table into its own second cache space;

[0008] S200, the pixel columns of the original image are padded according to the size of the receptive field, so that the columns of the original image are integer multiples of the receptive field to obtain the pixel values ​​of the padded image;

[0009] S300, set a basic cross pixel block, and move the basic cross pixel block column by column in the pixel values ​​of the completed image to obtain the pixel values ​​of the cross pixel block;

[0010] S400, execute a parallel program to use the pixel value of the cross pixel block as an index of the lookup table, and determine the pixel value for super-resolution from the lookup table in the second cache space according to the index;

[0011] S500 performs interpolation on the pixel values ​​used for super-resolution to obtain the pixel block after super-resolution of the center pixel of the cross pixel block;

[0012] S600, rearrange all the pixel blocks according to their positions to form a super-resolution image of the original image.

[0013] Secondly, the present invention provides an FPGA, wherein a hardware acceleration method for image super-resolution reconstruction based on in-memory indexing is executed on the FPGA.

[0014] Beneficial effects:

[0015] 1. This invention replaces the linear pixel block index with a cross-shaped pixel block and uses a high-resolution pixel block arrangement of r×r with rotation design in four directions: 0°, 90°, 180° and 270° to replace the rotation of the entire image. This simplifies the rotation operation of image adaptive enhancement, which is difficult to implement in hardware, and at the same time greatly saves the amount of data read and written, making it easier to allocate hardware resources more fully and rationally.

[0016] 2. In the hardware system of this invention, a parallel acceleration function is reasonably designed based on the hardware storage resources and the size of the lookup table (LUT). The image is divided into multiple blocks, and the LUT is read and calculated for each block of data to achieve parallel reading and calculation of multiple LUTs. The results are arranged according to the segmentation to finally obtain the HR image, which greatly reduces the processing speed.

[0017] 3. This invention fully designs a hardware acceleration system for SR-LUT and verifies its functionality and processing speed using a development board. Currently, the industry mainly deploys it on mobile devices such as smartphones, and due to limited resources, only functional verification is performed without corresponding hardware acceleration strategies, which fully demonstrates the advantages of this invention.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a hardware acceleration method for image super-resolution reconstruction based on in-memory indexing provided by the present invention.

[0020] Figure 2 This is the processing flowchart of the super-resolution module;

[0021] Figure 3 This is a schematic diagram illustrating the implementation of super-resolution using parallelized LUT reading;

[0022] Figure 4 It refers to the overall system architecture;

[0023] Figure 5 It is the internal data channel of SR-Core;

[0024] Figure 6 It is an image obtained using bilinear interpolation;

[0025] Figure 7 It is an image processed in a hardware system using the method of this invention. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0027] Before introducing the present invention, the technical concept of the present invention will be briefly described first.

[0028] This invention proposes a parallel hardware acceleration system based on a cross-index lookup table. The system first writes the original image data, then sends a start calculation signal after writing, and a completion calculation signal after completion. Upon receiving the signal, the result data is read out. This invention trains a network model with a fixed receptive field and passes the output values ​​of the learned deep model to a lookup table (LUT). During testing, it detaches from the convolutional neural network model, reads low-resolution (LR) image data on the hardware system, and then uses a cross-index module to query the corresponding HR image output values ​​of multiple LUTs in parallel. After rearranging, the final HR image is obtained.

[0029] This invention provides a hardware acceleration method for image super-resolution reconstruction based on in-memory indexing, applied on an FPGA. The core module of this invention, implemented on the FPGA, upsamples the original resolution data to obtain high-resolution data. The entire process is as follows: write the original image data; after writing, send a start signal; after calculation, send a done signal. Upon receiving the done signal, the result data is read out. The acceleration scheme of this invention is described in detail below.

[0030] refer to Figures 1 to 4 As shown, considering processing speed and power consumption, the hardware deployment of this algorithm often cannot simply implement basic computational logic. Therefore, this invention sets up a src ctrl module, a sr lut module, and a resctrl module within the FPGA; thus, this invention designs corresponding hardware acceleration circuits for all three modules. Specifically, the src ctrl module executes steps S100 to S300, the sr lut module executes steps S400 to S500, and the resctrl module executes step S600.

[0031] The srcctrl module of this invention mainly implements two functions:

[0032] ① The srcctrl module controls the writing and reading of raw data. To achieve the "ping-pong" operation, the srcctrl module internally uses two 32KB modules (128-bit data width, 2KB address depth). 11 RAM of size ).

[0033] ② The `srcctrl` module outputs the raw pixels of the cross-pixel block and transmits them to the next-level module `srlut` for super-resolution. This function is implemented by three sub-modules: `padding`, `srcmem`, and `linebuffer`. The `padding` module outputs the correct read enable and read address according to the padding rules. `srcmem` stores the raw data, and the finally read raw data is transmitted to the `linebuffer` module. After buffering the data, `linebuffer` outputs three lines of data. `crosspixels` converts the three lines of pixels into five pixels of the cross-pixel block.

[0034] The SR LUT module of this invention mainly implements super-resolution processing, the sampled-LUT mem submodule is used to store LUT data, and the tri-interpolation submodule restores the sampling error through interpolation. The processing flow is as follows: Figure 2 As shown, the specific process is as follows:

[0035] The input five cross-shaped pixel blocks are divided into four 1x2 pixel blocks in four directions. Triangular interpolation and lookup table operations are performed on each block. Then, the resulting 4x4 pixel blocks are rotated by a certain angle and added together to obtain the final high-resolution 4x4 pixel block. Since the pixels are distributed in four different rows, the storage addresses are not contiguous. Therefore, the address of each row is calculated before output.

[0036] The res ctrl module of this invention mainly controls the writing and reading of result data, while the res mem submodule is used to store result data.

[0037] like Figure 1 As shown, the hardware acceleration method for image super-resolution reconstruction based on in-memory indexing of the present invention includes:

[0038] S100, the FPGA reads the pixel values ​​of the original image and the lookup table from the SD card in three-line units, and places the pixel values ​​of the original image into the first cache space and the lookup table into its own second cache space (sample);

[0039] The lookup table (LUT) stores pixel values ​​used for super-resolution. The lookup table is set by uniformly sampling all pixel values ​​and is obtained using these values ​​and a trained convolutional neural network (CNN) model. In this step, the pixel values ​​are input into the trained CNN model, which then loads the weights of each layer to obtain the lookup table; the lookup table is then stored on an SD card.

[0040] It is worth noting that: All LUT index possibilities are enumerated based on the size and shape of the receptive field. If the receptive field size is m×n, then there are a total of 2... 8*m*n In this case, to save LUT size, let the scaling factor be s, and use a fixed interval W = 2. s A uniformly sampled LUT (Sampled-LUT) is used, so only 2 are needed. 8*m*n-s Adding just one more case reduces the size by 2 compared to the original complete LUT. s This reduces storage resources by 1 time, significantly saving them and facilitating increased parallelism in the hardware system, thus improving speed. The sampled results are then used as input, and the previously trained weights are substituted into the neural network from step one to obtain the LUT.

[0041] S200, the pixel columns of the original image are padded according to the size of the receptive field, so that the columns of the original image are integer multiples of the receptive field to obtain the pixel values ​​of the padded image;

[0042] S300, set a basic cross pixel block, and move the basic cross pixel block column by column in the pixel values ​​of the completed image to obtain the pixel values ​​of the cross pixel block;

[0043] S400, execute a parallel program to use the pixel value of the cross pixel block as an index of the lookup table, and determine the pixel value for super-resolution from the lookup table in the second cache space according to the index;

[0044] S500 performs interpolation on the pixel values ​​used for super-resolution to obtain the pixel block after super-resolution of the center pixel of the cross pixel block;

[0045] S600, rearrange all the pixel blocks according to their positions to form a super-resolution image of the original image.

[0046] In this invention, when obtaining the cross pixel block index, the linebuffer submodule in the srcctrl module pre-buffers two rows of data and outputs the three rows containing the cross pixel block index in sequence. crosspixels then converts the three rows of pixels into five pixels for the cross pixel block. Because two rows are pre-buffered, the first four pixels of the first two rows do not need to be accessed repeatedly. This allows one cross to be output in one clock cycle, whereas without buffered data, at least three clock cycles are required, significantly improving speed.

[0047] The `srlut` module transforms the four rotations of a linear pixel block into a cross-shaped pixel block index, eliminating the need for pixel rearrangement and omitting a tedious step. Furthermore, when performing super-resolution processing based on the cross-shaped pixel block index, the intermediate interpolation calculation requires three LUT lookups. This invention stores multiple LUTs for simultaneous invocation, and performs three LUT lookups concurrently during calculation, thus parallelizing the interpolation process and reducing overall computation time.

[0048] To implement the "ping-pong" operation, the resctrl module is designed with two 512KB modules (128-bit data width, 2KB address depth). 15 A RAM of 1.5 oz can simultaneously store the results of calculations and output the results of the previous calculation, thus speeding up the processing.

[0049] When performing super-resolution on a single image, the limited storage resources at the programmable logic (PL) end prevent the entire image from being super-resolutiond at once. Therefore, the image needs to be divided into multiple blocks for processing. This invention leverages the advantages of LUT storage to design an acceleration scheme for parallel processing of these blocks, such as... Figure 3 As shown, LR is processed by parallel reading of LUT to obtain SR. The final processing speed depends on the number of LUTs stored in the hardware system and the number of LUTs read in parallel.

[0050] As an optional embodiment of the present invention, the training process of the convolutional neural network model includes:

[0051] S01, Obtain the DIV2K dataset; the DIV2K dataset includes multiple images;

[0052] S02, multiple images are input as training samples into a convolutional neural network model with a fixed receptive field, and the convolutional neural network model is trained to obtain the weights of each convolutional layer, convolutional mapping layer, and upsampling layer; and the weights of the convolutional neural network are adjusted using a loss function and backpropagation.

[0053] The convolutional neural network model of the present invention includes a convolutional module and an activation module. The convolutional module includes a three-layer structure: the first layer is a convolutional layer with a kernel size equal to the receptive field, the second layer is a convolutional mapping layer, and the third layer is an upsampling layer. The activation module includes three activation functions, each of which is set between the layers of the convolutional module.

[0054] It is worth noting that because the network is limited by the receptive field size, a large number of convolutional layers will not infinitely improve accuracy, but will instead lead to faster convergence. Therefore, this invention utilizes shallow convolutional and activation modules to form the basic structure of the network, ensuring that the kernel size of the first layer is the same as the receptive field. The last layer employs a sub-pixel convolution (Pixelshuffle) module, which achieves upsampling by shifting pixels from the channel dimension to the length and width dimensions. Let PS be the output of Pixelshuffle, and the input size of the upsampling layer be (N, C*r*r, H, W), and the output size be (N, C, r*H, r*W). The specific operation is as follows:

[0055]

[0056] In the formula, T represents the input, h, w, and c represent the corresponding three-dimensional coordinates of the output, and r is the super-resolution magnification factor.

[0057] S03. Repeat step S02 until the training cutoff condition is met to obtain a trained convolutional neural network.

[0058] As an optional embodiment of the present invention, S02 includes:

[0059] S021, crop each image to obtain a cropped image;

[0060] This invention uses images cropped from the DIV2K dataset as input, passes them through a convolutional neural network with a fixed receptive field and adaptive enhancement, and then outputs HR images. The loss function is designed as follows:

[0061]

[0062] In the above formula, y i Let l represent the target image of the ground truth (GT), and l be the mean square error.

[0063] S022, the cropped image is input into a convolutional neural network model with a fixed receptive field, so that the convolutional neural network model rotates each cropped image by 90 degrees each time to obtain a rotation result, and the average of all rotation results is obtained to obtain a rotated image.

[0064] This invention also incorporates an adaptive enhancement operation during the training phase, which applies to the input image x. i Rotation enhancements of 90°, 180°, and 270° are performed. To preserve the integrity of image edge information, reflection filling is applied to the image. Then, an inverse enhancement operation is performed on the network output. The final output... This is the average of several anti-enhancement operations, specifically as follows:

[0065]

[0066] In the above formula, R j This indicates the operation of rotating the image to j×90 degrees. This represents the operation of rotating in the reverse direction to j×90 degrees, where f is the aforementioned convolutional neural network.

[0067] S023, Design a loss function based on the rotated image and the cropped image;

[0068] S024, Adjust the weights of the convolutional neural network model to reduce the loss function.

[0069] As an optional embodiment of the present invention, S400 of the present invention includes:

[0070] S410, the lookup table is placed at multiple addresses in its own second storage space (sample);

[0071] S420, a corresponding indexing program is set for each cross pixel block, and the indexing program corresponds to the address of a lookup table;

[0072] S430, each indexing program is executed in parallel according to the address, such that the indexing program uses the pixel value of its own cross pixel block as the index of its own lookup table, and determines the pixel value for super-resolution in its own lookup table according to the index.

[0073] This invention provides a hardware acceleration system for image super-resolution reconstruction based on in-memory indexing. The system is applied on an FPGA, which includes a src ctrl module, a srlut module, and a res ctrl module. The src ctrl module executes steps S100 to S300, the srlut module executes steps S400 to S500, and the res ctrl module executes step S600.

[0074] S100: Read the pixel values ​​of the original image and the lookup table from the SD card in three-line units, and place the pixel values ​​of the original image into the first cache space and place the lookup table into its own second cache space (sample);

[0075] S200, the pixel columns of the original image are padded according to the size of the receptive field, so that the columns of the original image are integer multiples of the receptive field to obtain the pixel values ​​of the padded image;

[0076] S300, set a basic cross pixel block, and move the basic cross pixel block column by column in the pixel values ​​of the completed image to obtain the pixel values ​​of the cross pixel block;

[0077] S400, execute a parallel program to use the pixel value of the cross pixel block as an index of the lookup table, and determine the pixel value for super-resolution from the lookup table in the second cache space according to the index;

[0078] S500 performs interpolation on the pixel values ​​used for super-resolution to obtain the pixel block after super-resolution of the center pixel of the cross pixel block;

[0079] S600, rearrange all the pixel blocks according to their positions to form a super-resolution image of the original image.

[0080] This invention provides an FPGA on which a hardware acceleration method for image super-resolution reconstruction based on in-memory indexing is executed.

[0081] The effectiveness of this invention can be further demonstrated through the following verification experiments.

[0082] This invention uses the AX7Z100 development board to verify the functionality of the designed super-resolution IP (SR-Core). The FPGA model of this development board is XC7Z100-2FFG900I, with a dual-core ARM Cortex-A9 core. The lookup table data (lutdata) is embedded within the srlut module. The overall hardware system architecture of this invention is as follows: Figure 4 As shown, the overall process is as follows:

[0083] 1. Under the control of ARM Cortex-A9, sampled-lut and raw image data are read from the SD card and transmitted to SR-Core via the AXI bus.

[0084] 2. Upon sending the start processing signal, SR-Core completes the super-resolution processing of the low-resolution image. SR-Core consists of three main modules: src ctrl, srlut, and res ctrl. The src ctrl module passes the original image data to the srlut module, which performs the super-resolution processing and passes the result data to the res ctrl module. Once processing is complete, a completion signal is sent.

[0085] 3. Finally, the results are written to the SD card via the AXI bus under the control of the ARM Cortex-A9. The SR-Core internal data channel is as follows: Figure 5 As shown.

[0086] For the verification of image performance metrics:

[0087] 1. Common Set5 and Set14 test sets are used as test images.

[0088] 2. This experiment uses a 4x super-resolution method and employs peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as evaluation metrics for image quality.

[0089] 3. The validation set was super-resolution processed using the traditional super-resolution methods bilinear interpolation and bicubic interpolation. The experimental results are shown in Table 1.

[0090] 4. The FSRCNN-s method was used to perform super-resolution processing on the validation set. The experimental results are shown in Table 1.

[0091] 5. The validation set was super-resolution processed using the method proposed in this invention. The experimental results are shown in Table 1.

[0092] Table 1 Comparison of Quantitative Parameters of Comparative Test Results

[0093]

[0094] From Table 1 and in combination Figure 6 and Figure 7 visible:

[0095] The peak signal-to-noise ratio (PSNR) and SSIM of the image after super-resolution processing using the method of this invention are significantly higher than those obtained using the two interpolation methods, and also have certain advantages over FSRCNN-s, indicating that the image processed by super-resolution using this invention retains more detailed and structural information.

[0096] The results in Table 1 fully demonstrate that the super-resolution processing of images using the method of the present invention yields better results, preserving more detailed information of the original scene and exhibiting better structure preservation characteristics.

[0097] Verification of hardware resource utilization and processing speed:

[0098] The resource utilization of the FPGA is shown in Table 2 below:

[0099] Table 2 Resource Utilization Rate

[0100]

[0101] As shown in Table 2, LUT resources account for only 7.39%, of which 53.49% is used for storing sampled-LUTs and 38.07% for multiply-accumulate logic calculations; DSP resources account for only 0.1%; and FF resources account for 1.67%. This demonstrates that the designed super-resolution IP requires minimal computational logic resources and involves a small computational load during super-resolution processing. BRAM utilization is 36.42%, of which approximately 98.54% is used for storing raw and result data, and approximately 0.36% is used for line cache.

[0102] Table 3 compares the computational cost, resource consumption, and time consumption of several common super-resolution methods. It can be seen that the indicators of the proposed method are superior to those of the bicubic interpolation method, which fully demonstrates that the proposed method has a significant speed advantage over general deep learning methods.

[0103] Table 3 Comparison of computational cost, resource consumption, and time consumption of several super-resolution methods

[0104]

[0105]

[0106] Ultimately, the results showed that this invention, running on an FPGA at a clock frequency of 100MHz, took approximately 34.8ms to enlarge a single low-resolution image (960×540×3) by 4×4 times (3840×2160×3). Compared to the currently mainstream FSRCNN method, this is approximately 10 times faster.

[0107] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0108] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0109] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A hardware acceleration method for image super-resolution reconstruction based on in-memory indexing, characterized in that, When applied to FPGAs, the hardware acceleration method for image super-resolution reconstruction based on in-memory indexing includes: S100: Read the pixel values ​​of the original image and the lookup table from the SD card in three rows, and place the pixel values ​​of the original image into the first cache space and place the lookup table into its own second cache space; S200, the pixel columns of the original image are padded according to the size of the receptive field, so that the columns of the original image are integer multiples of the receptive field to obtain the pixel values ​​of the padded image; S300, Set a basic cross pixel block, and move the basic cross pixel block column by column in the pixel values ​​of the completed image to obtain the pixel values ​​of the cross pixel block; the cross pixel block includes five pixels; S400, execute a parallel program, use the pixel value of the cross pixel block as the index of the lookup table, and determine the pixel value for super-resolution from the lookup table in the second cache space according to the index; S500 interpolates the pixel values ​​used for super-resolution to obtain the pixel block after super-resolution of the center pixel of the cross pixel block; S600, rearrange all the pixel blocks according to their positions to form a super-resolution image of the original image; The lookup table stores the pixel values ​​used for super-resolution, and the lookup table is obtained through the following steps: Set the value of all pixels by uniform sampling; The pixel values ​​are input into the trained convolutional neural network model, and the convolutional neural network model loads the weights of each layer to obtain the lookup table; Store the lookup table to the SD card; The training process of the convolutional neural network model includes: S01, Obtain the DIV2K dataset; the DIV2K dataset includes multiple images; S02, multiple images are input as training samples into a convolutional neural network model with a fixed receptive field, and the convolutional neural network model is trained to obtain the weights of each convolutional layer, convolutional mapping layer, and upsampling layer; and the weights of the convolutional neural network are adjusted using a loss function and backpropagation. S03, Repeat step S02 until the training cutoff condition is met to obtain a trained convolutional neural network; S02 includes: S021, crop each image to obtain a cropped image; S022, the cropped image is input into a convolutional neural network model with a fixed receptive field, so that the convolutional neural network model rotates each cropped image by 90 degrees each time to obtain a rotation result, and the average of all rotation results is obtained to obtain a rotated image. S023, Design a loss function based on the rotated image and the cropped image; S024, Adjust the weights of the convolutional neural network model to reduce the loss function.

2. The hardware acceleration method for image super-resolution reconstruction based on in-memory indexing according to claim 1, characterized in that, The FPGA includes: a src ctrl module, a sr lut module, and a res ctrl module; the src ctrl module executes steps S100 to S300, the sr lut module executes steps S400 to S500, and the res ctrl module executes step S600.

3. The hardware acceleration method for image super-resolution reconstruction based on in-memory indexing according to claim 1, characterized in that, The convolutional neural network model includes a convolutional module and an activation module. The convolutional module has a three-layer structure: the first layer is a convolutional layer with the same kernel size as the receptive field, the second layer is a convolutional mapping layer, and the third layer is an upsampling layer.

4. The hardware acceleration method for image super-resolution reconstruction based on in-memory indexing according to claim 3, characterized in that, The input size of the upsampling layer is (N, C) r r, H, W), output size is (N, C, r) H,r W), the specific operation is as follows: In the formula, T represents the input, h, w, and c represent the corresponding three-dimensional coordinates of the output, and r is the super-resolution magnification factor.

5. The hardware acceleration method for image super-resolution reconstruction based on in-memory indexing according to claim 1, characterized in that, The S400 includes: S410, the lookup table is placed at multiple addresses in its own second storage space; S420, a corresponding indexing program is set for each cross pixel block, and the indexing program corresponds to the address of a lookup table; S430, each indexing program is executed in parallel according to the address, such that the indexing program uses the pixel value of its own cross pixel block as the index of its own lookup table, and determines the pixel value for super-resolution in its own lookup table according to the index.

6. A hardware acceleration system for image super-resolution reconstruction based on in-memory indexing, characterized in that, Used to perform the hardware acceleration method for image super-resolution reconstruction based on in-memory indexing as described in any one of claims 1 to 5.

7. An FPGA, characterized in that, The hardware acceleration method for image super-resolution reconstruction based on in-memory indexing, as described in any one of claims 1 to 5, is executed on the FPGA.

Images