Real-time image super-resolution processing system and method based on FPGA

Abstract

The invention discloses a real-time image super-resolution processing system and method based on an FPGA, and relates to the technical field of image processing, and the system comprises at least one convolution module and a pixel alignment and rearrangement module. The convolution module obtains convolution input pixel data, edge pixel filling processing and caching processing are carried out on the convolution input pixel data to obtain a pixel matrix, convolution processing is carried out according to the pixel matrix to obtain convolution output pixel data, and data caching is carried out based on the block random access memory in the processing process of the convolution module; the pixel alignment and rearrangement module obtains to-be-aligned pixel data, alignment processing and rearrangement processing are carried out on the to-be-aligned pixel data, processed target output pixel data are obtained, and the to-be-aligned pixel data is convolution output pixel data output by the last convolution module; the resolution of the target output pixel data is higher than that of the to-be-processed image frame. In this way, the efficiency of image super-resolution processing is improved, and the real-time requirement is met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly relates to a real-time image super-resolution processing system and method based on FPGA. Background Art

[0002] Super-Resolution refers to the method of improving the resolution of the original image through hardware or software. The process of obtaining a high-resolution image from a series of low-resolution images is super-resolution reconstruction. At present, users have higher and higher requirements for the image quality of videos or images, while the high-resolution original sources are relatively limited. Therefore, image super-resolution processing is required.

[0003] In related technologies, image super-resolution processing can be performed based on a pre-constructed hardware system. For example, a Field Programmable Gate Array (FPGA) can be used for super-resolution processing. However, in related technologies, when using an FPGA-based hardware system for image super-resolution processing, it is usually necessary to introduce an external memory (DDR, Double Data Rate Synchronous Dynamic Random Access Memory) outside the FPGA for caching, and its access latency is relatively high, which affects the efficiency of image super-resolution processing and is difficult to meet the real-time requirements.

[0004] Therefore, related technologies need to be improved and developed. Summary of the Invention

[0005] The main purpose of this application is to provide a real-time image super-resolution processing system and method based on FPGA, aiming to solve the technical problems in related technologies that when using an FPGA-based hardware system for image super-resolution processing, it is necessary to introduce an external memory, the access latency is relatively high, which is not conducive to improving the efficiency of image super-resolution processing and is not conducive to meeting the real-time requirements of image processing.

[0006] To achieve the above purpose, the first aspect of this application provides a real-time image super-resolution processing system based on FPGA. Among them, the real-time image super-resolution processing system based on FPGA includes: At least one convolution module, and a pixel alignment and rearrangement module; The above convolution module is used to obtain convolution input pixel data, perform edge pixel padding processing and caching processing on the above convolution input pixel data to obtain a pixel matrix, and perform convolution processing based on the above pixel matrix to obtain convolution output pixel data. Among them, during the processing of the above convolution module, data caching is performed based on a block random access memory. If the above convolution module is the first convolution module, the above convolution input pixel data is the pixel data corresponding to the image frame to be processed. Otherwise, the above convolution input pixel data is the convolution output pixel data corresponding to the previous convolution module of the above convolution module; The above pixel alignment and rearrangement module is used to obtain the pixel data to be aligned, perform alignment processing and rearrangement processing on the above pixel data to be aligned, and obtain the processed target output pixel data. Among them, the above pixel data to be aligned is the convolution output pixel data output by the last convolution module, and the resolution of the above target output pixel data is higher than the resolution corresponding to the above image frame to be processed.

[0007] Optionally, the above convolution module includes a pixel padding processing sub-module and a convolution control sub-module; The above pixel padding processing sub-module is used to obtain convolution input pixel data, perform edge pixel padding processing and caching processing on the above convolution input pixel data, and output the pixel matrix obtained after processing to the above convolution control sub-module; The above convolution control sub-module is used to perform convolution processing on the above pixel matrix to obtain convolution output pixel data.

[0008] Optionally, the above system includes a plurality of cascaded convolution modules.

[0009] Optionally, in any convolution module, the above convolution input pixel data and the above convolution output pixel data have the same format.

[0010] Optionally, the above pixel padding processing sub-module includes a pixel padding unit and a row caching unit; The above pixel padding unit determines a target padding mode from a variety of padding modes according to preset configuration information, and performs edge pixel padding processing on the above convolution input pixel data according to the above target padding mode to obtain pixel padding data. Among them, the above variety of padding modes include zero padding, copy padding, and symmetric padding; The above row caching unit receives the above pixel padding data, performs pixel caching row by row on the above pixel padding data, and directly outputs the above pixel matrix.

[0011] Optionally, at least one convolution kernel unit and an activation layer are provided in the above convolution control sub-module; Each of the above convolution kernel units simultaneously performs a convolution operation according to the received pixel matrix to obtain at least one set of convolution data, where the convolution data corresponds one-to-one to the convolution kernel units, and the size of the pixel matrix is the same as the size of the convolution kernel units; The above activation layer performs a non-linear transformation on the above convolution data according to a preset activation function to obtain and output the above convolution output pixel data.

[0012] Optionally, the above pixel alignment and rearrangement module includes a pixel alignment sub-module and at least one cascaded pixel rearrangement sub-module; The above pixel alignment sub-module is used to perform alignment processing on the multi-channel above-mentioned pixel data to be aligned to obtain aligned pixel data, and input the above aligned pixel data into the first pixel rearrangement sub-module, where the number of channels of the above pixel data to be aligned matches the number of target convolution kernels, and the number of target convolution kernels is the number of convolution kernel units set in the convolution control sub-module of the last convolution module; The above pixel rearrangement sub-module is used to perform rearrangement processing on the received input data to obtain rearranged pixel data and output it to the next pixel rearrangement sub-module, where the rearranged pixel data output by the last pixel rearrangement sub-module is the target output pixel data, and the above target output pixel data is single-channel real-time video stream data.

[0013] Optionally, the number of the above pixel rearrangement sub-modules matches the number of the above target convolution kernels; The resolution magnification between the above target output pixel data and the above image frame to be processed matches the number of the above pixel rearrangement sub-modules and the number of the above target convolution kernels.

[0014] Optionally, the resolution corresponding to the above image frame to be processed is any one of a preset variety of resolutions; The frame rate of the video to which the above image frame to be processed belongs is any one of a preset variety of frame rates; The number of convolution modules in the above system can be adjusted to adjust the neural network structure corresponding to the above system.

[0015] A second aspect of the present application provides a method for real-time image super-resolution processing based on FPGA, where the method for real-time image super-resolution processing based on FPGA is applied to any one of the above real-time graphic super-resolution processing systems, and the method includes: Obtain the pixel data corresponding to the image frame to be processed; Perform at least one target operation based on the pixel data corresponding to the image to be processed, and obtain convolution output pixel data. The target operation includes: performing edge pixel padding processing and caching processing on the convolution input pixel data to obtain a pixel matrix, performing convolution processing based on the pixel matrix to obtain convolution output pixel data. If the target operation is the first target operation, the convolution input pixel data is the pixel data corresponding to the image frame to be processed; otherwise, the convolution input pixel data is the convolution output pixel data obtained after the previous target operation of the target operation. During the processing of the target operation, data caching is performed based on a block random access memory; Perform alignment processing and rearrangement processing on the convolution output pixel data obtained after the last target operation to obtain the processed target output pixel data, where the resolution of the target output pixel data is higher than the resolution of the image frame to be processed.

[0016] As can be seen from the above, in the solution of this application, a real-time image super-resolution processing system based on FPGA is provided. The system includes: at least one convolution module and a pixel alignment and rearrangement module. The convolution module is used to obtain convolution input pixel data, perform edge pixel padding processing and caching processing on the convolution input pixel data to obtain a pixel matrix, and perform convolution processing based on the pixel matrix to obtain convolution output pixel data. During the processing of the convolution module, data caching is performed based on a block random access memory. If the convolution module is the first convolution module, the convolution input pixel data is the pixel data corresponding to the image frame to be processed; otherwise, the convolution input pixel data is the convolution output pixel data corresponding to the previous convolution module of the convolution module. The pixel alignment and rearrangement module is used to obtain the pixel data to be aligned, perform alignment processing and rearrangement processing on the pixel data to be aligned, and obtain the processed target output pixel data, where the pixel data to be aligned is the convolution output pixel data output by the last convolution module, and the resolution of the target output pixel data is higher than the resolution of the image frame to be processed.

[0017] Compared with the prior art, the present application provides a new FPGA-based real-time image super-resolution processing system, including at least one convolutional module and a pixel alignment and rearrangement module. During the process of image super-resolution processing based on this FPGA-based real-time image super-resolution processing system, edge pixel filling processing, caching processing, and convolutional processing are performed based on the convolutional module, and then alignment processing and rearrangement processing are performed based on the pixel alignment and rearrangement module, so as to achieve image super-resolution processing and obtain target output pixel data with higher resolution. Moreover, during the processing of the convolutional module, data caching is directly performed based on the internal block random access memory provided by the FPGA, without introducing an external memory, which can reduce access latency, is beneficial to improving the efficiency of image super-resolution processing, and is beneficial to meeting the real-time requirements of image (or video frame) processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 is a schematic diagram of the application scenario of an FPGA-based real-time image super-resolution processing system provided by the present application; Figure 2 is a schematic diagram of the component modules of an FPGA-based real-time image super-resolution processing system provided by an embodiment of the present application; Figure 3 is a schematic diagram of the data processing flow of an FPGA-based real-time image super-resolution processing system provided by an embodiment of the present application; Figure 4 is a schematic diagram of the image super-resolution processing flow based on FPGA in the prior art provided by an embodiment of the present application; Figure 5 is a schematic diagram of the specific data processing flow of a pixel filling processing sub-module 11 provided by an embodiment of the present application; Figure 6 is a schematic diagram of the specific data processing flow of a convolutional control sub-module 12 provided by an embodiment of the present application; Figure 7 is a schematic diagram of a convolutional interface adaptation mechanism provided by an embodiment of the present application; Figure 8 is a schematic diagram of a traditional convolutional interface mechanism provided by an embodiment of the present application; Figure 9 is a schematic diagram of the specific processing flow of a pixel alignment and rearrangement module 2 provided by an embodiment of the present application; Figure 10 It is a schematic diagram of the specific processing flow of a pixel rearrangement sub-module 22 provided by an embodiment of the present application; Figure 11 It is a schematic diagram of the flow of a real-time image super-resolution processing method based on FPGA provided by an embodiment of the present application. Specific embodiments

[0020] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0021] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0022] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0023] It should be further understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in this specification and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to classifying" according to the context. Similarly, the phrases "if determined" or "if classified into [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once classified into [the described condition or event]", or "in response to classifying into [the described condition or event]" according to the context.

[0025] With reference to the accompanying drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0026] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application may be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present application. Therefore, the present application is not limited by the specific embodiments disclosed below.

[0027] With the development of video technology, users' expectations for video image quality are also getting higher and higher. Among them, the excellent user experience brought by 4K / 8K high resolution has made it one of the hotspots of concern in recent years. However, the original video sources with a resolution up to 8K are relatively limited, and the data volume of 8K videos is extremely large, which has an adverse impact on its popularization at multiple levels, from production sources, data storage to real-time transmission. With the gradual maturity of super-resolution algorithms, by means of algorithms and other methods, upscaling 4K-resolution videos to 8K resolution can significantly reduce the resource consumption of videos during the storage and transmission stages.

[0028] For the original 4K-resolution videos, traditional 2x super-resolution processing methods face many difficulties. When the software-based super-resolution algorithms are executed on general-purpose processors, due to their sequential execution architecture and complex computational tasks, such as a large number of matrix operations and convolution operations in deep learning algorithms, the processing speed is severely lagged and cannot meet the real-time requirements. For example, common software algorithms may take several seconds to process a frame of 4K video, resulting in obvious stuttering during video playback and greatly affecting the viewing experience.

[0029] In terms of hardware implementation, the existing solutions also have significant drawbacks. Although some dedicated chips can accelerate the processing, they are more targeted at smaller resolutions such as 1080p or lower, and it is difficult to achieve good processing effects for 2K and even 4K input resolutions. In addition, most hardware solutions are insufficient in terms of resource utilization and flexibility, and it is difficult to achieve efficient hardware resource configuration and rapid deployment while ensuring the super-resolution effect.

[0030] In an application scenario, image super-resolution processing can be performed based on a preset hardware system. For example, image super-resolution processing can be performed based on an FPGA. However, traditional FPGA video processing solutions usually cannot solely use programmable logic (PL) resources. Instead, they often rely on the processing system (PS) resources on the FPGA and utilize the network on chip (NOC) for PS / PL interaction. This not only increases the complexity and power consumption of the system but also may introduce additional latency. Moreover, an external memory needs to be introduced for caching, which has a high access latency and requires complex control and interface design, affecting the efficiency of image super-resolution processing and making it difficult to meet the real-time requirements.

[0031] To solve at least one of the above-mentioned multiple technical problems, a real-time image super-resolution processing system based on an FPGA is provided in this application. The system includes: at least one convolution module and a pixel alignment and rearrangement module. The convolution module is configured to obtain convolution input pixel data, perform edge pixel padding processing and caching processing on the convolution input pixel data to obtain a pixel matrix, and perform convolution processing based on the pixel matrix to obtain convolution output pixel data. During the processing of the convolution module, data caching is performed based on a block random access memory. If the convolution module is the first convolution module, the convolution input pixel data is the pixel data corresponding to the image frame to be processed. Otherwise, the convolution input pixel data is the convolution output pixel data corresponding to the previous convolution module of the convolution module. The pixel alignment and rearrangement module is configured to obtain pixel data to be aligned, perform alignment processing and rearrangement processing on the pixel data to be aligned, and obtain the processed target output pixel data. The pixel data to be aligned is the convolution output pixel data output by the last convolution module, and the resolution of the target output pixel data is higher than the resolution of the image frame to be processed.

[0032] Compared with the prior art, the present application provides a new real-time image super-resolution processing system based on FPGA, including at least one convolutional module and one pixel alignment and rearrangement module. During the process of image super-resolution processing based on this real-time image super-resolution processing system based on FPGA, edge pixel filling processing, caching processing and convolutional processing are performed based on the convolutional module, and then alignment processing and rearrangement processing are performed based on the pixel alignment and rearrangement module, so as to realize image super-resolution processing and obtain target output pixel data with higher resolution. And during the processing of the convolutional module, data caching is directly performed based on the internal block random access memory provided by the FPGA, without introducing an external memory, which can reduce access latency, is beneficial to improving the efficiency of image super-resolution processing, and is beneficial to meeting the real-time requirements of image (or video frame) processing.

[0033] Specifically, the real-time image super-resolution processing system based on FPGA in the present application is a system constructed and implemented based on FPGA, and can realize real-time video super-resolution processing by performing super-resolution processing on video frame images. It can accept real-time video streams with various resolutions and frame rates, such as 2K@30Hz, 4K@30Hz, 4K@120Hz, etc., perform two-fold super-resolution processing on the input video while keeping the frame rate unchanged, and then output a high-resolution real-time video stream to improve the video quality. At the same time, this super-resolution processing system supports the implementation of various internal neural network structures and can support the real-time processing of video streams with various resolutions and frame rates, and can adapt to different input resolutions.

[0034] Aiming at the actual problems that the 2K and 4K video data volumes are huge and have high requirements for processing speed, traditional methods are difficult to complete super-resolution processing within the specified time. The present application constructs an efficient FPGA processing architecture, fully exploits the hardware parallelism, ensures the real-time processing of video streams, eliminates stuttering and latency, and guarantees smooth playback. And in the present application, it gets rid of the dependence on the PS resources and DDR cache on the FPGA, and only relies on the PL resources and on-chip block random access memory (BRAM, Block Random Access Memory) throughout the process to implement the entire super-resolution processing flow.

[0035] Furthermore, in the present application, at least one convolutional module is provided in the real-time image super-resolution processing system based on FPGA. The convolutional module can be regarded as a modular designed module, with input and output interfaces that can be cascaded with each other, and can realize the convolutional plug-and-play function. And according to different super-resolution network structures and different FPGA resource situations, the convolutional units can be flexibly configured, so as to realize the adaptability to different input resolutions, different input frame rates, and different network structures.

[0036] Figure 1FIG. 0 is a schematic diagram of an application scenario of a real-time image super-resolution processing system based on FPGA provided by the present application. As shown in FIG. Figure 1 Figure 1 As shown, the real-time image super-resolution processing system based on FPGA provided by the present application can be used to perform real-time processing on a low-resolution video to obtain a high-resolution video. It should be noted that the above-mentioned real-time image super-resolution processing system based on FPGA can process videos or images. When processing videos, each video frame is processed correspondingly. Each frame image of the input low-resolution video stream enters the system frame by frame for super-resolution processing, and after alignment and rearrangement processing, a high-resolution video is output.

[0037] Figure 2 FIG. 6 is a schematic diagram of the constituent modules of a real-time image super-resolution processing system based on FPGA provided by an embodiment of the present application. As shown in FIG. Figure 2 Figure 2 As shown, an embodiment of the present application provides a real-time image super-resolution processing system based on FPGA. Specifically, the above system includes: At least one convolution module 1 and one pixel alignment and rearrangement module 2; The above convolution module 1 is used to obtain convolution input pixel data, perform edge pixel filling processing and caching processing on the above convolution input pixel data to obtain a pixel matrix, and perform convolution processing based on the above pixel matrix to obtain convolution output pixel data. Among them, during the processing of the above convolution module 1, data caching is performed based on a block random access memory. If the above convolution module 1 is the first convolution module 1, the above convolution input pixel data is the pixel data corresponding to the image frame to be processed. Otherwise, the above convolution input pixel data is the convolution output pixel data corresponding to the previous convolution module 1 of the above convolution module 1; The above pixel alignment and rearrangement module 2 is used to obtain the pixel data to be aligned, perform alignment processing and rearrangement processing on the above pixel data to be aligned, and obtain the target output pixel data after processing. Among them, the above pixel data to be aligned is the convolution output pixel data output by the last convolution module 1, and the resolution of the above target output pixel data is higher than the resolution corresponding to the image frame to be processed.

[0038] It should be noted that Figure 2m convolutional modules 1 are shown, but in actual use, only one convolutional module 1 can be set, which is not specifically limited here. The value of m can be set and adjusted according to actual needs, and is not specifically limited here either. In the embodiments of the present application, taking the above system including multiple cascaded convolutional modules 1 as an example for specific illustration, in each of the cascaded convolutional modules 1, the output of the previous convolutional module 1 is used as the input of the current convolutional module 1, and the output of the current convolutional module 1 is used as the input of the next convolutional module 1. It should be noted that the number of convolutional modules 1 in the system can be set and adjusted according to actual needs and the neural network architecture actually to be constructed. In the embodiments of the present application, the adjustment of the number of convolutional modules 1 can be determined based on adjusting the top-level configuration parameters of the system, so that the neural network structure corresponding to the system can be flexibly adjusted, and the plug-and-play of the convolutional module 1 can be realized. In the scenario where the number of convolution kernels needs to be adjusted, there is no need to change the structure of the entire convolutional module 1, and only the number of convolutional modules 1 needs to be flexibly adjusted, which is beneficial to improving the flexibility of the overall system configuration and expanding the applicable scenarios of the entire system.

[0039] In a specific application scenario, after the system receives the externally input pixel data, according to the top-level configuration parameters, adaptive padding processing is performed on the edge pixels, and the padded data is line-buffered. The pixel data after buffering will be output in the form of a pixel matrix. The size of the pixel matrix can be flexibly configured at the top level and is equal to the size of the subsequent convolution kernel. The pixel matrix will complete convolution operations in the convolutional module 1, including parallel convolution multiplication, pipelined addition, and activation operations. Finally, after passing through the pixel alignment and rearrangement module 2 for pixel rearrangement and alignment, a two-fold super-resolution result is output. The above top-level configuration parameters include input resolution, frame rate, network structure parameters, etc., which are not specifically limited here.

[0040] It should be further noted that the FPGA-based real-time image super-resolution processing system in the embodiments of the present application is implemented based on FPGA. During the use of the system, during the processing of the above convolutional module 1, data caching is performed based on the block random access memory (BRAM, Block Random Access Memory) provided by FPGA. The above block random access memory can be regarded as a part of the FPGA-based real-time image super-resolution processing system or a separate part outside the FPGA-based real-time image super-resolution processing system, which is not specifically limited here, as long as it satisfies that the convolutional module 1 and the block random access memory can communicate with each other during the data caching process to achieve data caching.

[0041] Figure 3 It is a schematic diagram of the data processing flow of an FPGA-based real-time image super-resolution processing system provided by the embodiments of the present application. Figure 4It is a schematic diagram of an image super-resolution processing flow based on FPGA in the prior art provided by an embodiment of the present application.

[0042] In the embodiment of the present application, as Figure 3 shown, the above convolution module 1 includes a pixel filling processing sub-module 11 and a convolution control sub-module 12; wherein, the above pixel filling processing sub-module 1 is used to obtain convolution input pixel data, perform edge pixel filling processing and caching processing on the above convolution input pixel data, and output the pixel matrix obtained after processing to the above convolution control sub-module 12; the above convolution control sub-module 12 is used to perform convolution processing on the above pixel matrix to obtain convolution output pixel data.

[0043] In the full FPGA processing flow proposed by the present application, the caching of pixel data is implemented through the on-chip BRAM inside the FPGA, without using the external memory of the FPGA. All controls and operations are processed by the PL in the FPGA, without using the PS+PL+DDR architecture commonly used in traditional image processing systems (the processing process of traditional image processing systems is as Figure 4 shown), giving full play to the advantage of parallel processing of the PL resources in the FPGA, realizing full pipelining of pixel processing (Fully Pipelined), greatly reducing the demand for external memory (such as DDR memory) in the traditional processing flow, and only using a small amount of BRAM resources to achieve real-time processing of 4K video. And no additional resources are used in the full process, making it easy to be transplanted to ASIC for implementation.

[0044] Figure 5 It is a schematic diagram of the specific data processing flow of the pixel filling processing sub-module 11 provided by an embodiment of the present application. As Figure 5 shown, the above pixel filling processing sub-module 11 includes a pixel filling unit 111 and a row caching unit 112; the above pixel filling unit 111 determines the target filling mode from multiple filling modes according to the preset configuration information, and performs edge pixel filling processing on the above convolution input pixel data according to the above target filling mode to obtain pixel filling data, wherein the above multiple filling modes include zero filling, copy filling and symmetric filling; the above row caching unit 112 receives the above pixel filling data, performs pixel caching row by row on the above pixel filling data, and directly outputs the above pixel matrix.

[0045] It should be noted that the pixel filling processing sub-module 11 uses BRAM to complete all caching functions, giving full play to the low access latency, initialization-free, and low-cost characteristics of BRAM. The pixel filling unit 111 therein can dynamically adjust the filling mode according to the configuration, such as zero filling, symmetric filling, copy filling, etc. The caching sub-module receives the filled pixel data and caches it pixel by pixel. The line caching unit 112 directly outputs the pixel matrix required for the convolution operation according to the configuration, and the size of the pixel matrix is the size of the subsequent convolution kernel, without additional pixel stitching logic. The pixel filling processing sub-module 11 can adapt to different input resolutions and frame rates. In a specific application scenario, three input modes are integrated inside the pixel filling processing sub-module 11: single-pixel input per clock (adapting to 4K@30Hz), dual-pixel input per clock (adapting to 4K@60Hz), and quadruple-pixel input per clock (adapting to 4K@120Hz). Multiple input modes are adapted through a unified counting and comparison mechanism, including filling and caching functions for different resolutions.

[0046] It should be further noted that Figure 5 where h and w respectively represent the number of rows and columns of the input pixels. In the embodiments of the present application, the pixel filling unit 111 fills one row or one column at each of the upper, lower, left, and right edges of the input pixels, but the actual filling method and the actual number of filled rows and columns are not specifically limited.

[0047] Figure 6 is a schematic diagram of the specific data processing flow of a convolution control sub-module 12 provided by the embodiments of the present application. As Figure 6 shown, at least one convolution kernel unit 121 and an activation layer 122 are provided in the above-mentioned convolution control sub-module 12; each of the above-mentioned convolution kernel units 121 simultaneously performs a convolution operation according to the received above-mentioned pixel matrix to obtain at least one set of convolution data, wherein the above-mentioned convolution data corresponds one-to-one with the above-mentioned convolution kernel unit 121, and the size of the above-mentioned pixel matrix is the same as the size of the above-mentioned convolution kernel unit; the above-mentioned activation layer 122 performs a non-linear transformation on the above-mentioned convolution data according to a preset activation function to obtain and output the above-mentioned convolution output pixel data.

[0048] Specifically, based on the huge advantages of FPGA for parallel processing, multiple convolution kernel units 121 can be integrated within the convolution control sub-module 12, which can simultaneously execute multiple sets of convolution operations. The convolution control sub-module 12 can integrate n convolution kernel units 121, where the value range of n is 1 - 16 (specifically limited by FPGA resources and can be set and adjusted according to actual requirements). Multiple multipliers are integrated inside each convolution kernel unit 121, which can simultaneously execute multiple convolution multiplication operations. In this way, the latency of convolution processing can be reduced, the speed of convolution operations can be accelerated, and full pipeline processing of convolution operations can be achieved. The above convolution data corresponds one-to-one with the above convolution kernel units 121, that is, one set of convolution data is generated after being processed by one convolution kernel unit 121.

[0049] A variety of non-linear activation functions, such as LeakyRelu, Relu, etc., can be integrated within the above activation layer 122, and the activation operation switch and its parameters can be freely controlled to adapt to various network structures. Specifically, the LeakyRelu activation function is shown in the following formula (1):

[0050] Where, represents the input data, is a hyperparameter, and its specific value can be determined and adjusted according to actual requirements.

[0051] The Relu activation function is shown in the following formula (2):

[0052] It should be noted that other non-linear activation functions can also be integrated within the activation layer 122, and which activation function is specifically called can be determined by adjusting the top-level configuration parameters, which are not specifically limited herein.

[0053] Furthermore, within any one convolution module 1, the above convolution input pixel data and the above convolution output pixel data have the same format. Thus, it can be ensured that the data between cascaded convolution modules 1 can be directly processed without additional format adaptation processing.

[0054] Figure 7 is a schematic diagram of a convolution interface adaptation mechanism provided by an embodiment of the present application, Figure 8 is a schematic diagram of a traditional convolution interface mechanism provided by an embodiment of the present application. By comparing Figure 7 and Figure 8It can be seen that in the traditional FPGA convolution implementation, even if the structure of row buffer combined with convolution is adopted, the input and output are in different formats, and an additional format adaptation module is required to achieve the cascade of convolution modules. In the embodiments of the present application, the output of the pixel filling processing sub-module 11 matches the input of the convolution control sub-module 12, and the pixel filling processing sub-module 11 and the convolution control sub-module 12 are combined into a convolution module 1. The output pixels and the input pixels of the convolution module 1 also maintain the same pattern. In this way, the interface adaptation mechanism can ensure that the input and output of the convolution module 1 maintain the same structure (such as single pixel per clock, double pixels per clock, four pixels per clock, etc.), so as to directly cascade multiple convolution modules 1 with each other, adapt to various network structures, various input resolutions and frame rates, and this process can be achieved only through simple configuration.

[0055] Figure 9 FIG. 4 is a schematic diagram of the specific processing flow of a pixel alignment and rearrangement module 2 provided by an embodiment of the present application. As Figure 9 shown, the above-mentioned pixel alignment and rearrangement module 2 includes a pixel alignment sub-module 21 and at least one cascaded pixel rearrangement sub-module 22; the above-mentioned pixel alignment sub-module 21 is used to perform alignment processing on the multi-channel above-mentioned pixels to be aligned to obtain aligned pixel data, and input the above-mentioned aligned pixel data into the first pixel rearrangement sub-module 22. Among them, the number of channels of the above-mentioned pixels to be aligned matches the number of target convolution kernels, and the above-mentioned number of target convolution kernels is the number of convolution kernel units 121 set in the convolution control sub-module 12 of the last convolution module 1; the above-mentioned pixel rearrangement sub-module 22 is used to perform rearrangement processing on the received input data to obtain rearranged pixel data and output it to the next pixel rearrangement sub-module 22. Among them, the rearranged pixel data output by the last pixel rearrangement sub-module 22 is the target output pixel data, and the above-mentioned target output pixel data is single-channel real-time video stream data.

[0056] It should be noted that Figure 9 only one pixel rearrangement sub-module 22 is shown in FIG. 4. In actual use, multiple pixel rearrangement sub-modules 22 can be set according to actual needs, and each pixel rearrangement sub-module 22 is cascaded (that is, the output data of the previous pixel rearrangement sub-module 22 is used as the input data of the next pixel rearrangement sub-module 22). Specifically, the number of the above-mentioned pixel rearrangement sub-modules 22 matches the number of the above-mentioned target convolution kernels. The resolution magnification between the above-mentioned target output pixel data and the above-mentioned image frame to be processed matches the number of the above-mentioned pixel rearrangement sub-modules 22 and the number of the above-mentioned target convolution kernels.

[0057] The resolution corresponding to the to-be-processed image frame is any one of a plurality of preset resolutions; the frame rate of the video to which the to-be-processed image frame belongs is any one of a plurality of preset frame rates; the number of the convolutional modules 1 in the system can be adjusted to implement the adjustment of the neural network structure corresponding to the system. Among them, the plurality of preset resolutions and the plurality of preset frame rates can be set and adjusted according to actual requirements. That is, the system can support different input resolutions and different input frame rates (the corresponding video can be directly input and the video frames of the video can be processed separately), and the system includes at least one convolutional module 1 with an adjustable number, having the adaptability to different network structures.

[0058] Specifically, the pixel alignment sub-module 21 receives the multi-channel pixel data after convolutional processing and performs multi-channel pixel alignment to ensure that the pixels at the same spatial position in different channels can be correctly combined, avoiding image distortion or blurring. Then, the aligned multi-channel pixel data enters the pixel rearrangement sub-module 22.

[0059] Specifically, the number of channels of the to-be-aligned pixel data input to the first pixel rearrangement sub-module 22 matches the number of target convolution kernels, and the number of target convolution kernels is the number of convolution kernel units 121 set in the convolution control sub-module 12 of the last convolutional module 1. As Figure 9 shown, the number of target convolution kernels is 4. Correspondingly, the pixel alignment sub-module 21 aligns the pixel data of the input 4 channels and outputs the aligned pixel data of 4 channels to the pixel rearrangement sub-module 22. At this time, only one pixel rearrangement sub-module 22 is set to rearrange the pixel data and output it. And at this time, the resolution magnification between the target output pixel data and the to-be-processed image frame is 2 times, that is, 2-fold super-resolution is achieved.

[0060] In an application scenario, if the number of target convolution kernels is 16, two pixel rearrangement sub-modules 22 are set. The 16-channel pixel data is input to the pixel alignment sub-module 21 for alignment to obtain the aligned 16-channel pixel data, and then input to the first pixel rearrangement sub-module 22 for rearrangement to obtain the rearranged 4-channel pixel data, and then input to the second pixel rearrangement sub-module 22 for rearrangement to obtain the rearranged single-channel pixel data. At this time, the super-resolution is 4 times. The adjustment of the number of pixel rearrangement sub-modules 22 and the specific processing flow are analogized accordingly and will not be elaborated here.

[0061] Figure 10 is a schematic diagram of the specific processing flow of a pixel rearrangement sub-module 22 provided by an embodiment of the present application. As Figure 10As shown, the pixel rearrangement sub-module 22 rearranges the multi-channel pixel data according to certain rules and stitches them into a single-channel, high-resolution real-time video stream. This process can convert multi-channel data into single-channel data with high spatial resolution while maintaining information integrity, avoiding information loss, and the pixel rearrangement process requires no complex calculations and consumes few resources. It should be noted that the specific pixel rearrangement rules in the pixel rearrangement sub-module 22 can be set and adjusted according to actual needs, and no specific limitations are imposed here. In the embodiment of this application, taking 4-channel input as an example, the set pixel rearrangement rule is that the inputs of the first and second channels will be alternately stitched into the first row of output; the inputs of the third and fourth channels will be alternately stitched into the second row of output; the subsequent input-output rules are the same as above, but this is not a specific limitation.

[0062] Specifically, in the real-time image super-resolution processing system based on FPGA provided in the embodiment of this application, m convolution modules 1 can be integrated, corresponding to m independent convolutional layers. The minimum value of m is 1, and the maximum value is determined by the on-chip BRAM resources. The number of m can be flexibly configured at the top layer. This system can support video streams with multiple input resolutions and input frame rates, and can be flexibly configured through the top layer.

[0063] Only the input resolution, frame rate, and network structure parameters need to be configured at the top layer of the system. In a specific application scenario, the relevant configuration parameters are as follows: Input resolution configuration: 4K (3840*2160), 1080p (1920*1080), and 720p (1280*720). Input frame rate configuration: 24 frames, 30 frames, 60 frames, and 120 frames. Network structure configuration: One-channel input one-channel output convolutional layer, one-channel input four-channel output convolutional layer, four-channel input four-channel output convolutional layer, four-channel input one-channel output convolutional layer, one-channel input sixteen-channel output convolutional layer, sixteen-channel input one-channel output convolutional layer, four-channel input sixteen-channel output convolutional layer, sixteen-channel input four-channel output convolutional layer, non-linear activation function layer, etc. The input and output interfaces of the above network structures are the same and can be connected in any combination.

[0064] In an embodiment of the present application, the above system is also tested based on an actual application scenario. Specifically, an HDMI input video stream signal is used to receive pixel data frame by frame and perform convolution processing. The number of convolution layers required can be flexibly configured according to actual needs. After pixel alignment and rearrangement, a video stream with 2x super resolution is output through the HDMI interface. A super resolution network is constructed with a neural network structure of m = 2 (two convolution modules 1) and n = 4 (four convolution kernel units 121 integrated in each convolution control sub-module 12). The train part of the DIV8K dataset is used as the training set. The high-resolution images in the training set are downsampled to obtain low-resolution images; the original high-resolution images are used as labels, and the low-resolution images are used as inputs to form training samples (HR / LR image pairs). The adaptive moment estimation optimization algorithm (Adam) and the cosine annealing warm restart learning rate algorithm are used for training, and total_iter is 100000. Using the above super resolution algorithm network, when the input is a 4K@30Hz video stream that conforms to the CTA-861 standard, only 921 Kb of BRAM resources are required to achieve real-time super resolution effects, and the input-to-output processing delay is only 0.127 ms. If Figure 4 the traditional FPGA processing flow shown is adopted, the input-to-output processing delay is approximately 124.692 ms, 4K@30Hz real-time processing cannot be achieved, and a large amount of external DDR resources need to be consumed.

[0065] As can be seen from the above, the embodiment of the present application provides a new FPGA-based real-time image super resolution processing system, including at least one convolution module 1 and one pixel alignment and rearrangement module 2. During the process of performing image super resolution processing based on this FPGA-based real-time image super resolution processing system, edge pixel filling processing, caching processing, and convolution processing are performed based on the convolution module 1, and then alignment processing and rearrangement processing are performed based on the pixel alignment and rearrangement module 2, so as to achieve image super resolution processing and obtain target output pixel data with higher resolution. During the processing of the convolution module 1, data caching is directly performed based on the internal block random access memory provided by the FPGA, without introducing external memories, which can reduce access latency, is beneficial to improving the efficiency of image super resolution processing, and is beneficial to meeting the real-time requirements of image (or video frame) processing.

[0066] Furthermore, in the embodiments of the present application, the traditional Advanced RISC Machine (ARM) architecture is abandoned, and only the FPGA architecture is used to give full play to its parallel processing ability. Without relying on external complex hardware, super-resolution processing with flexible configuration of input resolution and frame rate can be achieved, and the effect of real-time super-resolution processing of 4K@120Hz video can be realized. Moreover, the dependence on DDR cache is eliminated, and on-chip BRAM is used to implement all data caching functions. This greatly shortens the data transmission path, reduces the transmission delay, and is easy to perform timing constraints and control. Without using additional resources, it is convenient for ASIC transplantation. The pixel filling processing sub-module 11 can adapt to a variety of different input resolutions and filling modes. And this module can flexibly modify the size of the output pixel matrix according to the configuration to adapt to the needs of subsequent convolution kernels. The pixel filling processing sub-module 11 and the convolution control sub-module 12 are combined to form an independent convolution module 1. The input and output interfaces are mutually adapted and can be cascaded with each other to implement a variety of different neural network structures, and the control logic is simple. Only simple configuration is required to implement a variety of structures without changing the code. In this way, using the above FPGA-based real-time image super-resolution processing system, two-fold real-time super-resolution processing can be achieved only using the FPGA core and on-chip BRAM resources without using ARM and external DDR cache, and it supports a variety of input resolutions and frame rates, such as 2K@30Hz, 4K@30Hz, 4K@120Hz, etc. At the same time, the convolution module 1 in the above system can be plug-and-play, and a variety of neural network structures can be realized by simple configuration without much modification of the code.

[0067] As Figure 11 shown in, corresponding to the above FPGA-based real-time image super-resolution processing system, the embodiments of the present application further provide an FPGA-based real-time image super-resolution processing method applied to the above system. The above method includes the following steps: Step S100, obtaining pixel data corresponding to the image frame to be processed; Step S200, performing at least one target operation according to the pixel data corresponding to the image to be processed to obtain convolution output pixel data, where the target operation includes: performing edge pixel filling processing and caching processing on the convolution input pixel data to obtain a pixel matrix, performing convolution processing according to the pixel matrix to obtain convolution output pixel data. If the target operation is the first target operation, the convolution input pixel data is the pixel data corresponding to the image frame to be processed, otherwise, the convolution input pixel data is the convolution output pixel data obtained after the previous target operation of the target operation. During the processing of the target operation, data caching is performed based on a block random access memory; Step S300: Align and rearrange the convolution output pixel data obtained after the last target operation to obtain the processed target output pixel data, where the resolution of the above-mentioned target output pixel data is higher than the resolution of the above-mentioned image frame to be processed.

[0068] It should be noted that the above method is executed and implemented based on the above-mentioned FPGA-based real-time image super-resolution processing system. Specifically, a convolution module 1 in the above system is used to perform a target operation once.

[0069] Specifically, the pixel data corresponding to the above-mentioned image to be processed is used as the convolution input pixel data of the first convolution module 1 to trigger the first convolution module 1 to perform edge pixel filling processing and caching processing on the above-mentioned convolution input pixel data to obtain a pixel matrix, perform convolution processing on the basis of the above-mentioned pixel matrix to obtain convolution output pixel data, and use the above-mentioned convolution output pixel data as the convolution input pixel data corresponding to the next convolution module 1, where data caching is performed based on a block random access memory during the processing of the above-mentioned convolution module 1; Use the convolution output pixel data output by the last convolution module 1 as the pixel data to be aligned and input it into the pixel alignment and rearrangement module 2 to trigger the pixel alignment and rearrangement module 2 to perform alignment processing and rearrangement processing on the above-mentioned pixel data to be aligned to obtain the processed target output pixel data, where the resolution of the above-mentioned target output pixel data is higher than the resolution of the above-mentioned image frame to be processed.

[0070] In this way, during the processing, data caching is directly performed based on the internal block random access memory provided by the FPGA, without introducing an external memory, which can reduce the access latency, is beneficial to improving the efficiency of image super-resolution processing, and is beneficial to meeting the real-time requirements of image (or video frame) processing.

[0071] It should be noted that the specific implementation processes of each step in the above-mentioned FPGA-based real-time image super-resolution processing method can refer to the specific descriptions in the above-mentioned FPGA-based real-time image super-resolution processing system, and will not be elaborated here.

[0072] It should be noted that the division method of each module of the above-mentioned FPGA-based real-time image super-resolution processing system is not unique, and is not specifically limited here.

[0073] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0074] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be assigned to different functional units and modules according to needs, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the foregoing method embodiments and will not be elaborated here.

[0075] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0076] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0077] In the embodiments provided in this application, it should be understood that the disclosed system / terminal device and method can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For example, the above-mentioned division of modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0078] If the above integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above embodiment methods of this application, it can also be completed by instructing relevant hardware through a computer program. The above computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the above computer program includes computer program code, and the above computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The above computer-readable medium can include: any entity or device that can carry the above computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, and software distribution medium, etc. It should be noted that the content included in the above computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction.

[0079] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not deviate from the spirit and scope of the technical solutions of this application for the corresponding technical solutions, and should all be included within the protection scope of this application.

Claims

1. A real-time image super-resolution processing system based on FPGA, characterized in that, The system includes: at least one convolution module, and a pixel alignment and rearrangement module; The convolution module is used to obtain convolution input pixel data, perform edge pixel padding processing and caching processing on the convolution input pixel data to obtain a pixel matrix, and perform convolution processing on the pixel matrix to obtain convolution output pixel data. Wherein, during the processing of the convolution module, data caching is performed based on a block random access memory. If the convolution module is the first convolution module, the convolution input pixel data is the pixel data corresponding to the image frame to be processed. Otherwise, the convolution input pixel data is the convolution output pixel data corresponding to the previous convolution module of the convolution module; The pixel alignment and rearrangement module is used to obtain pixel data to be aligned, perform alignment processing and rearrangement processing on the pixel data to be aligned, and obtain the processed target output pixel data. Wherein, the pixel data to be aligned is the convolution output pixel data output by the last convolution module, and the resolution of the target output pixel data is higher than the resolution of the image frame to be processed.

2. The real-time image super-resolution processing system based on FPGA according to claim 1, characterized in that The convolution module includes a pixel padding processing sub-module and a convolution control sub-module; The pixel padding processing sub-module is used to obtain convolution input pixel data, perform edge pixel padding processing and caching processing on the convolution input pixel data, and output the obtained pixel matrix to the convolution control sub-module; The convolution control sub-module is used to perform convolution processing on the pixel matrix to obtain convolution output pixel data.

3. The real-time image super-resolution processing system based on FPGA according to claim 1, wherein, A plurality of cascaded convolution modules are included in the system.

4. The real-time image super-resolution processing system based on FPGA according to claim 3, characterized in that, In any convolution module, the formats of the convolution input pixel data and the convolution output pixel data are the same.

5. The real-time image super-resolution processing system based on FPGA according to claim 2, wherein The pixel padding processing sub-module includes a pixel padding unit and a row cache unit; The pixel padding unit determines a target padding mode from multiple padding modes according to preset configuration information, and performs edge pixel padding processing on the convolution input pixel data according to the target padding mode to obtain pixel padding data. Wherein, the multiple padding modes include zero padding, copy padding, and symmetric padding; The row cache unit receives the pixel padding data, performs pixel caching row by row on the pixel padding data, and directly outputs the pixel matrix.

6. The real-time image super-resolution processing system based on FPGA according to claim 5, wherein, At least one convolution kernel unit and an activation layer are provided in the convolution control sub-module; Each convolution kernel unit simultaneously performs a convolution operation according to the received pixel matrix to obtain at least one set of convolution data. Wherein, the convolution data corresponds one-to-one with the convolution kernel unit, and the size of the pixel matrix is the same as the size of the convolution kernel unit; The activation layer performs a non-linear transformation on the convolution data according to a preset activation function to obtain and output the convolution output pixel data.

7. The real-time image super-resolution processing system based on FPGA according to claim 6, wherein, The pixel alignment and rearrangement module includes a pixel alignment sub-module and at least one cascaded pixel rearrangement sub-module; The pixel alignment sub-module is used to align the multi-channel pixel data to be aligned to obtain aligned pixel data, and input the aligned pixel data into the first pixel rearrangement sub-module. Among them, the number of channels of the pixel data to be aligned matches the number of target convolution kernels, and the number of target convolution kernels is the number of convolution kernel units set in the convolution control sub-module of the last convolution module; The pixel rearrangement sub-module is used to rearrange the received input data to obtain rearranged pixel data and output it to the next pixel rearrangement sub-module. Among them, the rearranged pixel data output by the last pixel rearrangement sub-module is the target output pixel data, and the target output pixel data is single-channel real-time video stream data.

8. The real-time image super-resolution processing system based on FPGA according to claim 7, wherein The number of pixel rearrangement sub-modules matches the number of target convolution kernels; The resolution magnification between the target output pixel data and the image frame to be processed matches the number of pixel rearrangement sub-modules and the number of target convolution kernels.

9. The real-time image super-resolution processing system based on FPGA according to any one of claims 1 to 8, characterized in that, The resolution corresponding to the image frame to be processed is any one of a plurality of preset resolutions; The frame rate of the video to which the image frame to be processed belongs is any one of a plurality of preset frame rates; The number of convolution modules in the system can be adjusted to adjust the neural network structure corresponding to the system.

10. A real-time image super-resolution processing method based on FPGA, characterized in that, The method is applied to the FPGA-based real-time image super-resolution processing system according to any one of claims 1 to 9. The method includes: Obtain the pixel data corresponding to the image frame to be processed; According to the pixel data corresponding to the image to be processed, perform at least one target operation to obtain convolution output pixel data. Among them, the target operation includes: performing edge pixel filling processing and caching processing on the convolution input pixel data to obtain a pixel matrix, performing convolution processing on the pixel matrix to obtain convolution output pixel data. If the target operation is the first target operation, the convolution input pixel data is the pixel data corresponding to the image frame to be processed. Otherwise, the convolution input pixel data is the convolution output pixel data obtained after the previous target operation of the target operation. During the processing of the target operation, data caching is performed based on a block random access memory; Perform alignment processing and rearrangement processing on the convolution output pixel data obtained after the last target operation to obtain the processed target output pixel data. Among them, the resolution of the target output pixel data is higher than the resolution of the image frame to be processed.

Images