Image processing method based on FPGA, storage medium and computer program product

Abstract

The invention discloses an image processing method based on an FPGA, a storage medium and a computer program product, and relates to the technical field of semiconductor manufacturing. The method comprises the following steps: dividing and expanding a wafer image in a memory of an FPGA according to a convolution effective size and an edge effect size, and determining a target effective matrix and a target reading range of the target effective matrix; and obtaining read data according to the target reading range, and obtaining filling data for filling the target effective matrix located at the edge of the wafer image while obtaining the read data. And performing convolution preprocessing on the acquired data according to the pixel data groups corresponding to the target effective matrixes to obtain target data for executing image processing. Based on the pipeline design of caching the wafer image in the memory, positioning the target reading range and obtaining the filling data during reading, the occupation of the cache space of the memory is reduced, the processing delay of the wafer image is reduced, and the overall image processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of semiconductor manufacturing, and particularly relates to an FPGA-based image processing method, a storage medium, and a computer program product. BACKGROUND

[0002] In the field of semiconductor manufacturing, high-precision real-time detection of a wafer surface is crucial for chip yield control. For example, in wafer defect detection, by accurately and in real time detecting tiny defects on the wafer surface, process deviations in the semiconductor manufacturing process can be discovered in a timely manner, so that the equipment parameters of the production line can be adjusted, the chip yield can be improved, and the production cost can be reduced.

[0003] At present, high line frequency acquisition devices such as time delay integration (TDI) cameras are generally used to obtain wafer images by scanning the wafer, and then complex image processing of the wafer images is performed by a graphics processing unit (GPU). However, due to the memory architecture limitations and calculation mode characteristics of the GPU, the wafer image processing efficiency is poor. SUMMARY

[0004] The embodiments of the present application provide an FPGA-based image processing method, a storage medium, and a computer program product, which can reduce wafer image processing delay and improve overall image processing efficiency.

[0005] In a first aspect, the embodiments of the present application provide an FPGA-based image processing method applied to an FPGA, comprising:

[0006] According to a preset convolution effective size, a wafer image in a memory of the FPGA is divided into a plurality of initial effective matrices, and each initial effective matrix is expanded according to a preset edge effect size to obtain a target effective matrix corresponding to each initial effective matrix;

[0007] For each target effective matrix, a target reading range of the target effective matrix in the memory is determined according to a matrix coordinate of the target effective matrix in an image coordinate system of the wafer image and a matrix size of the target effective matrix;

[0008] Reading data of each target effective matrix is obtained from the memory according to the target reading range, and in the process of obtaining the reading data, padding data for padding the target effective matrix located at an edge of the wafer image is obtained according to the edge effect size;

[0009] The read data and the padding data are respectively subjected to convolution preprocessing according to pixel data groups corresponding to each of the target effective matrices to obtain target data, and the target data is sent to a graphics processing unit for image processing.

[0010] In a second aspect, the embodiment of the present application provides an FPGA-based image processing device, which comprises:

[0011] A matrix position module is configured to divide a wafer image in a memory of the FPGA into a plurality of initial effective matrices according to a preset convolution effective size, and expand each of the initial effective matrices according to a preset edge effect size to obtain a target effective matrix corresponding to each of the initial effective matrices.

[0012] An image traversal module is configured to determine, for each of the target effective matrices, a target reading range of the target effective matrix in the memory according to a matrix coordinate of the target effective matrix in an image coordinate system of the wafer image and a matrix size of the target effective matrix.

[0013] A memory reading module is configured to acquire read data of each of the target effective matrices from the memory according to the target reading range, and acquire padding data for padding the target effective matrix located at an edge of the wafer image according to the edge effect size during the acquisition of the read data.

[0014] A convolution preprocessing module is configured to respectively perform convolution preprocessing on the read data and the padding data according to pixel data groups corresponding to each of the target effective matrices to obtain target data, and send the target data to a graphics processing unit for image processing.

[0015] In a third aspect, the embodiment of the present application provides an FPGA-based image processing electronic device, which comprises a memory and a program or instruction stored in the memory and executable on a processor, and the program or instruction is executed by the processor to implement the FPGA-based image processing method provided in any one of the aspects of the embodiment of the present application.

[0016] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a program or instruction, and the program or instruction is executed by a processor to implement the FPGA-based image processing method provided in any one of the aspects of the embodiment of the present application.

[0017] In a fifth aspect, the embodiment of the present application provides a computer program product, and instructions in the computer program product are executed by a processor of an electronic device to enable the electronic device to perform the FPGA-based image processing method provided in any one of the aspects of the embodiment of the present application.

[0018] The technical scheme provided by the embodiment of the application brings at least the following beneficial effects:

[0019] In the image processing method based on FPGA provided by the embodiment of the application, first, the wafer image in the memory of the FPGA is divided and expanded according to the convolution effective size and the edge effect size, the target effective matrix and the target reading range of the target effective matrix are determined. Second, the reading data is obtained according to the target reading range, and at the same time the reading data is obtained, the padding data located at the edge of the wafer image is obtained. Finally, the reading data and the padding data are respectively preprocessed according to the pixel data group corresponding to each target effective matrix, and the target data for the image processing unit to execute image processing is obtained. Based on the pipeline design of caching the wafer image in the memory of the FPGA, positioning the target reading range and obtaining the padding data when reading, the occupation of the cache space of the memory is reduced, the processing delay of the wafer image is reduced, and the image processing efficiency of the wafer image is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical scheme of the embodiment of the application, the drawings needed to be used in the embodiment of the application will be briefly introduced below. For those skilled in the art, other drawings can also be obtained without creative labor on the premise.

[0021] Figure 1 is a flowchart of an image processing method based on FPGA provided by an embodiment of the application;

[0022] Figure 2 is a schematic diagram of the image data format transmitted by the TDI camera provided by an embodiment of the application;

[0023] Figure 3 is a schematic diagram of dividing the wafer image into a plurality of initial effective matrices based on the convolution effective size provided by an embodiment of the application;

[0024] Figure 4 is a schematic diagram of expanding the initial effective matrix to obtain the target effective matrix based on the edge effect size provided by an embodiment of the application;

[0025] Figure 5 is a schematic diagram of horizontal padding and vertical padding provided by an embodiment of the application;

[0026] Figure 6 is a schematic diagram of obtaining reading data from the memory provided by an embodiment of the application;

[0027] Figure 7This is a schematic diagram of horizontal padding based on the split transmission data packets provided in one embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of an FPGA-based image processing system provided in one embodiment of this application;

[0029] Figure 9a This is a schematic diagram of the vertical filling process provided in one embodiment of this application;

[0030] Figure 9b This is a schematic diagram of the bottom vertical filling process provided in one embodiment of this application;

[0031] Figure 9c This is a schematic diagram of the top vertical filling process provided in one embodiment of this application;

[0032] Figure 9d This is a schematic diagram of the process for obtaining pixel data when vertical fill is not triggered, provided in one embodiment of this application;

[0033] Figure 10 This is a schematic diagram of the structure of an FPGA-based image processing device provided in one embodiment of this application;

[0034] Figure 11 This is a schematic diagram of an FPGA-based image processing device provided in one embodiment of this application. Detailed Implementation

[0035] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0037] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0038] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0039] A Time Delay Integration Camera (TDI Camera) is a special type of linear scan camera, often used in high-speed, high-sensitivity continuous imaging scenarios (such as wafer inspection, flat panel display inspection, etc.).

[0040] A graphics processing unit (GPU) is a processor specifically designed for processing graphics and parallel computing tasks. In wafer inspection, GPUs are often used for complex image processing algorithms such as image denoising, edge detection, and feature extraction.

[0041] A Field-Programmable Gate Array (FPGA) is a parallel hardware architecture used for programmable hardware chips in real-time signal processing.

[0042] The effective size of convolution refers to the image size that can be processed in each single convolution operation during convolution preprocessing.

[0043] Edge effect size refers to the additional pixel width that needs to be extended for edge regions during convolution preprocessing to ensure that the boundary regions can also be fully convolved.

[0044] An image coordinate system is a two-dimensional coordinate system that represents the position of pixels in an image. It is usually defined by the top left corner as the origin, with the x-axis pointing to the right (column direction) and the y-axis pointing downwards (row direction).

[0045] Matrix height refers to the number of pixels in the row direction of the target effective matrix in the wafer image.

[0046] The target read range refers to the address range required to read the target valid matrix from the FPGA's memory.

[0047] Convolutional preprocessing refers to the convolutional operations performed on the wafer image by the FPGA before the wafer image data is sent to the GPU for complex image processing. These convolutional operations typically include image denoising, edge enhancement, and feature extraction.

[0048] Secondly, in the semiconductor manufacturing field, high-precision real-time inspection of wafer surfaces is crucial for chip yield control. For example, in wafer defect detection, by accurately and in real-time detecting minute defects on the wafer surface, process deviations in the semiconductor manufacturing process can be detected immediately, thereby adjusting the equipment parameters of the production line, improving chip yield, and reducing production costs.

[0049] Currently, high-line-frequency acquisition devices such as Time Delay Integration (TDI) cameras are generally used to scan the wafer and acquire wafer images. These images are then processed by a Graphics Processing Unit (GPU). However, due to the limitations of GPU memory architecture and computational characteristics, wafer image processing efficiency is poor.

[0050] To address the aforementioned technical problems, this application provides an FPGA-based image processing method, storage medium, and computer program product. In the FPGA-based image processing method provided in this application embodiment, the FPGA is responsible for the front-end convolutional preprocessing of the wafer image and the data management of the wafer image, while the GPU is responsible for the back-end image processing of the reconstructed target data, forming a heterogeneous computing architecture of FPGA plus GPU.

[0051] First, the FPGA divides the wafer image cached in its memory based on the effective convolution size to obtain an initial effective matrix. Second, to avoid edge effects in subsequent convolution preprocessing, the FPGA expands each initial effective matrix based on the edge effect size to obtain a target effective matrix. This determines the target read range of the target effective matrix. When acquiring data according to this target read range, padding data for filling the target effective matrix located at the edges of the wafer image is obtained based on the edge effect size, resulting in low memory usage. Finally, the FPGA performs convolution preprocessing on the acquired read data and padding data to obtain the target data for image processing by the graphics processing unit. Because the FPGA completes image segmentation, edge expansion, edge padding, and convolution preprocessing at the front end to obtain convolutional wafer image data, it avoids direct GPU processing of the wafer image, reducing computational burden. Furthermore, the FPGA possesses hardware-level parallel processing capabilities and low-latency data flow control capabilities. Through a pipelined design of real-time wafer image reception, caching, segmentation, and convolution preprocessing, the processing latency of the wafer image is reduced, and the image processing efficiency of the wafer image is improved.

[0052] For example, the FPGA-based image processing method provided in this application can be applied to the production line of a semiconductor manufacturing company for high-precision real-time wafer defect detection. In practical applications, wafer images obtained by scanning the wafer using devices such as TDI cameras are received in real time and written into the FPGA's memory. The wafer image in the memory is then divided, expanded, and filled. Read data is obtained according to the target read range of the expanded target effective matrix, as well as filling data for filling the target effective matrix located at the edges of the wafer image. The FPGA performs convolution preprocessing (such as edge detection and image denoising) on ​​the acquired read data and filling data to obtain target data. This target data is stored in a data storage device (such as an independent, centralized BRAM block in the FPGA). The GPU can then perform various complex image processing based on the target data to achieve high-precision real-time wafer defect detection.

[0053] It should be noted that the application scenarios described in the above embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. The FPGA-based image processing method provided in the embodiments of this application can be applied to various application scenarios that require convolution preprocessing and real-time image processing.

[0054] The following describes specific embodiments of an FPGA-based image processing method, apparatus, electronic device, storage medium, and computer program product provided in this application. First, an FPGA-based image processing method is introduced, which is applied to an FPGA.

[0055] Figure 1 This is a schematic flowchart illustrating an FPGA-based image processing method provided in an embodiment of this application. Figure 1 As shown, the method includes steps S100 to S103.

[0056] S100: Based on the preset effective convolution size, the wafer image in the FPGA memory is divided into multiple initial effective matrices, and each initial effective matrix is ​​expanded according to the preset edge effect size to obtain the target effective matrix corresponding to each initial effective matrix.

[0057] In one or more embodiments of this application, in order to utilize the limited resources of the FPGA to achieve real-time convolutional preprocessing of the wafer image in subsequent steps, and to avoid edge effects during the convolutional preprocessing, this application requires, in a step, to divide and expand the wafer image in the FPGA memory according to a preset effective convolutional size to obtain a target effective matrix.

[0058] Specifically, this application divides the wafer image in the FPGA memory into multiple initial effective matrices according to a preset convolution effective size, and expands each initial effective matrix according to a preset edge effect size to obtain the target effective matrix corresponding to each initial effective matrix.

[0059] It should be noted that this application does not limit the method of acquiring wafer images in memory, and can be set according to actual needs. In one or more embodiments of this application, the row data stream of the wafer image after scanning the wafer can be transmitted row by row and written into the memory of the FPGA. Of course, this application does not limit the type of device for scanning the wafer, and can be set according to actual needs, such as a TDI camera, a line scanning electron microscope, or a panel scanning camera that can transmit wafer images row by row. That is, regardless of whether the imaging sensor is a linear array, a panel array, or a TDI structure, as long as the device's external data transmission interface transmits the wafer image to the FPGA in the form of a "row-by-row continuous" row data stream, it falls within the protection scope of this application. This application does not limit the specific type of memory, and can be set according to actual needs, such as DDR memory configured for the FPGA, such as DDR4 memory, or on-chip BRAM.

[0060] Taking a TDI camera with a line scan frequency of 500K*9072 as an example, if the image format is Mono8 (8 bits = 1 pixel) and the image size is 9072*2048, the data transmission interface between the TDI camera and the FPGA uses a CXP interface or a 40G / 100G optical port (256-bit width). That is, a single wafer image consists of 9072 columns * 2048 rows of pixels. The FPGA receives the row data stream of the wafer image transmitted by the TDI camera, with each single wafer image including 2048 rows of data, and each row including 284 256-bit transmission data packets (tdata).

[0061] like Figure 2 The diagram illustrates a TDI camera image data transmission format according to an embodiment of this application. Continuing with the previous example, in this figure, the transmission data packet (tdata) 200 has an image capture format of 8 bits per pixel. Since a transmission data packet is 256 bits, a transmission data packet is 32 pixels, and each row includes 284 256-bit transmission data packets. tuser is the synchronization code 201, which includes a start-of-line (SOL) flag, an end-of-line (EOL) flag, a start-of-frame (SOF) flag, and an end-of-frame (EOF) flag. The start-of-frame and end-of-frame flags are used to identify the start and end of the wafer image. `tkeep` is a 202-bit double-word (DW) validity mask used to identify data validity. `tkeep` includes 8 flag bits, one of which identifies the validity of a single DW in `tdata`. `8'hFF` indicates that all 8 DWs are valid, i.e., all 256 bits are valid; `8'h0F` indicates that the lower 4 DWs are valid, i.e., 128 bits are valid. Lines 0 to 2047 represent the row index.

[0062] Meanwhile, since convolution is essentially a sliding window operation, meaning the output of each pixel in the wafer image depends only on its neighborhood information, this application can reduce latency and improve efficiency by partitioning the wafer image before convolution preprocessing. Furthermore, the hardware parallel structure of FPGAs is more suitable for parallel computation in small areas, and partitioning can improve the utilization of FPGA resources. Since convolution is an operation where the central pixel depends on neighborhood information, the effective convolution size of the input image must be larger than the convolution kernel size during convolution preprocessing. To avoid edge effects, the edges of the input image also need to be expanded and / or padded. Since expansion is performed within the wafer image without additional data, while padding requires additional padding data, this application divides the process of avoiding edge effects into two steps: expansion and padding, to reduce memory usage. In this step, only the initial effective matrix is ​​expanded.

[0063] Continuing with the previous example, since only 128 bits of the 284th tdata in each row are valid, the valid_matrix_size of the convolution can be set to 48. Based on this valid convolution size, the wafer image can be divided into 9072 / 48 = 189 horizontally and 2048 / 4 ≈ 43 vertically. That is, a single 9072*2048 wafer image can be divided into 189*43 initial valid matrices, including 189*42 48*48 initial valid matrices and 189*1 48*32 initial valid matrices. Convolution is an operation where the center pixel depends on neighborhood information, so generally the kernel size kernel_size_x = kernel_size_y and is odd. Assuming the kernel size is 17, then based on this kernel size of 17, the determined edge effect size should be (17-1) / 2 = 8. To avoid edge effects in subsequent convolution preprocessing, this application divides the wafer image into initial effective matrices based on the effective convolution size. Then, at the edges of each initial effective matrix, the image is expanded according to the edge effect size to obtain the target effective matrix corresponding to each initial effective matrix. In this application, the dimensions are described using the form "column direction (x-axis) dimension * row direction (y-axis) dimension." This form is not a specific technical detail of the FPGA-based image processing method implemented in this application and does not impose any limitations on the dimension-related content.

[0064] like Figure 3 The diagram illustrates how a wafer image is divided into multiple initial effective matrices based on the effective convolution size, according to an embodiment of this application. In this diagram, the effective convolution size is 48 pixels, the wafer image size is 9072*2048, and the division results in 189*43 initial effective matrices 300, including 189*42 48*48 initial effective matrices and 189*1 48*32 initial effective matrices.

[0065] This application expands each initial effective matrix according to a preset edge effect size to obtain the target effective matrix corresponding to the initial effective matrix, as follows:

[0066] valid_matrix_size_sum=valid_matrix_size_x*valid_matrix_size_y (1)

[0067] matrix_size_x=valid_matrix_size_x+i(kernel_size_x-1) / 2 (2)

[0068] matrix_size_y=valid_matrix_size_y +j(kernel_size_ y -1) / 2 (3)

[0069] Where `valid_matrix_size_sum` is the size of the initial valid matrix; `valid_matrix_size_x` is the matrix width in the column direction; and `valid_matrix_size_y` is the matrix height in the row direction. `matrix_size_x` is the width of the target valid matrix in the column direction; and `matrix_size_y` is the height of the target valid matrix in the column direction. `kernel_size_x * kernel_size_y` is the convolution kernel size. When at the edge, `i / j` is 1, indicating expansion only on one edge in the column / row direction; when in the middle, `i / j` is 2, indicating expansion on both edges in the column / row direction.

[0070] like Figure 4The diagram illustrates how an initial effective matrix is ​​expanded based on the edge effect size to obtain a target effective matrix, according to an embodiment of this application. Continuing with the previous example, the edge effect size is 8 pixels, and each initial effective matrix (48*48 or 48*32 pixels) is expanded. The initial effective matrices located at the top left and top right corners of the wafer image are expanded to obtain a target effective matrix of 56*56; the initial effective matrices located at the bottom left and bottom right corners of the wafer image are expanded to obtain 56*40; the initial effective matrices located at non-vertices of the wafer image and adjacent to the top edge are expanded to obtain a target effective matrix of 64*56; the initial effective matrices located at non-vertices of the wafer image and adjacent to the bottom edge are expanded to obtain a target effective matrix of 64*40 (401), such as regions of the wafer image that are non-vertices, adjacent to the bottom edge, and have filling patterns; the initial effective matrices located at non-vertices of the wafer image and adjacent to the left and right edges are expanded to obtain a target effective matrix of 56*64, such as regions of the wafer image that are non-vertices, adjacent to the right edge, and have filling patterns; the initial effective matrices located at non-edges of the wafer image are expanded to obtain a target effective matrix of 64*64, such as regions of the wafer image with filling patterns in the middle.

[0071] S101: For each of the target effective matrices, determine the target read range of the target effective matrix in the memory based on the matrix coordinates of the target effective matrix in the image coordinate system of the wafer image and the matrix size of the target effective matrix.

[0072] In one or more embodiments of this application, in order to obtain the read data in the memory in the form of a target effective matrix in subsequent steps, this application needs to determine the target read range of the target effective matrix in this step.

[0073] Specifically, for each target effective matrix, this application determines the target read range of the target effective matrix in the memory based on the matrix coordinates of the target effective matrix in the image coordinate system of the wafer image and the matrix size of the target effective matrix.

[0074] It should be noted that the process of determining the target read range in this application is the process of converting the coordinates of the matrix into cache addresses in memory. A single wafer image includes multiple rows of data. The process of writing the row data stream into memory in step S100 is performed on a unit basis for a single wafer image, that is, each wafer image in the memory is stored as a two-dimensional data structure. Therefore, in order to obtain the read data according to the target effective matrix in subsequent steps, this application needs to determine the target read range of each target effective matrix. Of course, this application does not limit the specific method of determining the target read range, and it can be set according to actual needs. In one or more embodiments of this application, this application can determine the target read range of the target effective matrix based on the matrix coordinates and matrix size of the target effective matrix, as follows:

[0075] First, for each target valid matrix, this application determines the pixel row coordinates of the first and last rows in the image coordinate system of the wafer image, based on the matrix row coordinates of the target valid matrix in the image coordinate system and the matrix height of the target valid matrix. Second, this application determines the read start address of the target valid matrix in memory based on the pixel row coordinates corresponding to the first row of the target valid matrix and the starting base address of the wafer image in memory. Finally, this application determines the read end address of the target valid matrix in memory based on the pixel row coordinates corresponding to the last row of the target valid matrix, the starting base address of the wafer image in memory, and the matrix width of the target valid matrix, and determines the target read range of the target valid matrix in memory based on the read start address and the read end address.

[0076] In this embodiment, matrix coordinates refer to the index of the target effective matrix in the image coordinate system of the wafer image after the wafer image is divided. For example, if the index of a target effective matrix is ​​1 in the column direction and 1 in the row direction, then the matrix coordinates of the target effective matrix are (1, 1). In this application, matrix coordinates are a two-dimensional coordinate system indexed by matrices, and pixel coordinates are a two-dimensional coordinate system indexed by pixels. The starting base address of the wafer image refers to the starting address where the wafer image is stored in memory. That is, each row of data in the wafer image is stored with an offset based on the starting base address. In other words, the row data stream received in real time in step S100 is written to the memory. In this memory, the wafer image is a two-dimensional data structure stored with row-by-row offset based on the starting base address. Therefore, the read start address is the address of the first pixel in the first row of the target effective matrix in memory, and the read end address is the address after the last pixel in the last row of the target effective matrix.

[0077] S102: Obtain read data for each target effective matrix from the memory according to the target read range, and during the process of obtaining the read data, obtain fill data for filling the target effective matrix located at the edge of the wafer image according to the edge effect size.

[0078] In one or more embodiments of this application, in order to avoid edge effects in subsequent convolution preprocessing steps, the target effective matrix can be used as a unit to obtain read data. This allows for the acquisition of read data of the target effective matrix located at the edge of the wafer image, while simultaneously acquiring fill data for filling the target effective matrix, thereby avoiding edge effects caused by the target effective matrix located at the edge.

[0079] Specifically, this application obtains read data of each target effective matrix from the memory according to the target read range, and in the process of obtaining read data, obtains fill data for filling the target effective matrix located at the edge of the wafer image according to the edge effect size.

[0080] It should be noted that this application does not limit the specific content of the acquired padding data, and can be set according to actual needs, such as edge copy padding, zero padding, and mirror padding. In one or more embodiments of this application, this application can automatically detect and fill the target effective matrix located at the edge of the wafer image while acquiring and reading the data, so as to meet the needs of subsequent convolution preprocessing, as follows:

[0081] First, this application can mark the target valid matrix corresponding to the currently read transmission data packet as the current read matrix, and then detect whether the current read matrix is ​​located at the edge of the wafer image while acquiring the read data of the current read matrix.

[0082] Secondly, this application determines the target column coordinates of the first or last column in the current read matrix in the image coordinate system, and obtains filling data for lateral filling when the target column coordinates are the same as the start or end column coordinates in the wafer image. Lateral filling includes determining a lateral filling region adjacent to the first or last column in the current read matrix in the wafer image based on the edge effect size, and filling within that lateral filling region.

[0083] Finally, this application determines the target row coordinates in the image coordinate system of the first or last row of the current read matrix. If the target row coordinates are the same as the starting or ending row coordinates in the wafer image, fill data for vertical filling is obtained. Vertical filling includes determining a vertical fill region in the wafer image that is adjacent to the first or last row of the current read matrix based on the edge effect size, and filling the vertical fill region.

[0084] likeFigure 5 The diagram shown is a schematic of horizontal and vertical fill provided in one embodiment of this application. Continuing with the previous example, in this figure, the horizontal fill region 500 and the vertical fill region 501 are determined according to the edge effect size. That is, the area with a size equal to the edge effect size extending outward from the edge of the wafer image is the fill region. The horizontal fill region refers to the fill region of the current read matrix in the column direction, and the vertical fill region refers to the fill region of the current read matrix in the row direction.

[0085] Furthermore, this application can acquire data with a width / height equal to the edge effect size located at the edge of the wafer image, which is used as padding data for mirror padding to avoid edge effects in subsequent convolution preprocessing. In one or more embodiments of this application, since the row data stream is received as a row of multiple transmission data packets (e.g., if the TDI camera's external interface is 256 bits, then one transmission data packet is 256 bits), after horizontal padding is triggered, this application can split the transmission data packets, and split the first and last transmission data packets in the row data stream to extract sub-data packets of the edge effect size for horizontal padding, as follows:

[0086] First, this application can split the transmission data packets located at the beginning and end of each row in the current read matrix based on the edge effect size to obtain a sequence of sub-data packets. The sub-data packet sequence includes at least a start sub-data packet at the beginning of the sequence and a stop sub-data packet at the end of the sequence, and the size of the start and stop sub-data packets is the edge effect size. Second, when the target column coordinates corresponding to the current read matrix are the same as the start column coordinates in the wafer image, this application copies the start sub-data packet as padding data for mirror filling in the horizontal padding area. Finally, when the target column coordinates are the same as the stop column coordinates in the wafer image, this application copies the stop sub-data packet as padding data for mirror filling in the horizontal padding area.

[0087] It should be noted that after obtaining the padding data for mirroring the horizontal padding area, the padding data needs to be filled into the horizontal padding area. In this embodiment, the application can copy the starting sub-data packet and use it as padding data to mirror-fill the horizontal padding area adjacent to the first column of the current read matrix; or copy the ending sub-data packet and use it as padding data to mirror-fill the horizontal padding area adjacent to the last column of the current read matrix.

[0088] After vertical fill is triggered, this application can determine the starting or ending copy range of the current read matrix based on the edge effect size, and obtain the data within the starting or ending copy range for vertical fill, as follows:

[0089] First, when the target row coordinates are the same as the starting row coordinates in the wafer image, this application determines the starting copy range of the current read matrix based on the edge effect size, and copies the data within the starting copy range as fill data for mirror filling in the vertical fill area. Second, when the target row coordinates are the same as the ending row coordinates in the wafer image, this application determines the ending copy range based on the edge effect size and the matrix height of the current read matrix, and copies the data within the ending copy range as fill data for mirror filling in the vertical fill area.

[0090] It should be noted that after obtaining the fill data for mirroring the vertical fill area, the fill data needs to be filled into the vertical fill area. In this embodiment, this application can copy the data within the initial copy range as fill data, and mirror this fill data into the vertical fill area adjacent to the first row of the current read matrix; similarly, it can copy the data within the final copy range as fill data, and mirror this fill data into the vertical fill area adjacent to the last row of the current read matrix. Continuing with the previous example, if the edge effect size is 8 pixels, then the initial copy range is the first 8 rows of the current read matrix. Therefore, the data from these first 8 rows is used as fill data and mirrored to fill the vertical fill area above the current read matrix. Similarly, if the final copy range is the last 8 rows of the current read matrix, then the data from these last 8 rows is used as fill data and mirrored to fill the vertical fill area below the current read matrix.

[0091] S103: The read data and the filling data are preprocessed by convolution according to the pixel data groups corresponding to each target effective matrix to obtain target data, and the target data is sent to the graphics processing unit that performs image processing.

[0092] In one or more embodiments of this application, the read data and fill data obtained in step S103 can be preprocessed by convolution, so that the preprocessed data can be sent to the graphics processing unit for image processing of the wafer image.

[0093] Specifically, this application can perform convolution preprocessing on the read data and padding data obtained in step S103 according to the pixel data group corresponding to each target effective matrix to obtain target data, and send the target data to the graphics processing unit that performs image processing.

[0094] It should be noted that the convolution preprocessing in this step is a process of convolution calculation based on the initial effective matrix and convolution kernel in the wafer image. To avoid edge effects, this application performs edge expansion and edge padding for each initial effective matrix. That is, in step S102, after obtaining the padding data for each target effective matrix, the padding data needs to be filled into the horizontal / vertical padding area at the edge of the target effective matrix. Then, for each target effective matrix, this application can use the read data of the target effective matrix and its corresponding padding data as the pixel data group of that target effective matrix. This application can perform convolution preprocessing on each pixel data group to obtain the target data for GPU image processing. To improve the real-time performance and efficiency of image processing, this application adopts a heterogeneous computing structure of front-end FPGA and back-end GPU. Furthermore, this application does not pad the initial effective matrix in memory or modify the wafer image data stored in memory. Instead, it directly obtains the padding data (e.g., for mirror padding, data located at the edge of the wafer image needs to be obtained) while obtaining the read data, thereby avoiding edge effects, reducing data acquisition latency, and improving efficiency.

[0095] Furthermore, this application does not limit the specific format and method of sending the target data to the graphics processing unit. It can be set according to actual needs. For example, the FPGA can be directly connected to the GPU to transmit the target data to the GPU; or the FPGA can be used as a terminal device and connected to the host computer (including the GPU) through the Peripheral Component Interconnect Express (PCIE) slot on the motherboard. In this case, after the FPGA performs convolution preprocessing on the wafer image, it can repackage the target data obtained after convolution preprocessing into a row data stream format and send it to the host computer through the PCIE interface so that the GPU can perform image processing based on the target data.

[0096] In the above method, this application is based on a pipeline design that caches wafer images in the memory of the FPGA, locates the target reading range, and obtains filling data during reading. This reduces the cache space occupied by the memory, reduces the processing latency of the wafer image, and improves the image processing efficiency of the wafer image.

[0097] In step S100, the wafer image needs to be divided based on the effective convolution size. There may be cases where the wafer image cannot be divided evenly by the effective convolution size. In such cases, the initial effective matrix at the horizontal or vertical division end is smaller than the expected matrix size (effective convolution size * effective convolution size). Therefore, in one or more embodiments of this application, in step S100, the wafer image needs to be divided based on the effective convolution size. If the size of the initial effective matrix at the horizontal and / or vertical division end is smaller than the effective convolution size, the initial effective matrix at the division end is taken as the terminal effective matrix, and the horizontal and / or vertical differences between the size of the terminal effective matrix and the effective convolution size are determined. Among the initial effective matrices that are horizontally and / or vertically adjacent to the terminal effective matrix in the wafer image and the terminal effective matrix, matrix pixels are selected based on the horizontal and / or vertical differences. These matrix pixels are then concatenated with the terminal effective matrix. Based on the concatenated terminal effective matrix, subsequent convolution preprocessing is performed. Afterward, the calculation results of the division end are read according to the actual position of the terminal effective matrix, avoiding the need to process initial effective matrices of other sizes separately. In other words, in this embodiment, the initial valid matrix at the end of the partition can still be used for subsequent convolution preprocessing according to the effective convolution size. There is no need to design separate control logic for the initial valid matrix at the end of the partition, so as to improve the efficiency of convolution preprocessing by reducing the complexity of logic design.

[0098] In addition, due to factors such as the burst transfer mechanism or memory control access of DDR memory, the read data or padding data obtained from the memory is not continuous. Therefore, before performing convolution preprocessing, the obtained padding data and read data can be cached in a first-in-first-out (FIFO) cache unit. Thus, according to the FIFO principle, the operation matrix data after each initial valid matrix expansion and padding can be continuously obtained in the FIFO cache unit. In one or more embodiments of this application, the read data and padding data obtained in step S103 can be stored in the FIFO cache unit, and then the read data and padding data can be obtained from the FIFO cache unit based on the size of each initial valid matrix expansion and / or padding operation matrix.

[0099] In addition, this application does not limit the specific method for determining the effective convolution size. It can be set according to actual needs. For example, the convolution kernel size can be selected based on the limitation of computing resources, and then based on real-time requirements (such as a TDI bandwidth of 400K*9072 pixels and a processing speed of 250M*14.175 pixels that the FPGA should meet), the actual processing speed of image processing under different effective convolution sizes with the same convolution kernel can be compared, and the effective convolution size with the higher actual processing speed can be selected. Of course, since the method of this application is applied to FPGA, such as in the FPGA of an image acquisition card, the computing resources of different image acquisition cards are different. Therefore, when comparing and selecting the effective convolution size, the variable of the image acquisition card can be considered.

[0100] It should be noted that if the kernel size in this application is 17*17, then comparisons can be made with effective convolutional sizes of 16*16, 48*16, 48*48, and 112*112, comparing resource consumption ratios, the difference between real-time processing speed and requirements, etc. When the effective convolutional size is 48*48, the resource consumption ratio is 0.6 and the difference between real-time processing speed and requirements is 5.04, making it the fastest compared to other effective convolutional sizes. Therefore, this application can choose an effective convolutional size of 48*48. The comparison and verification process is detailed in Table 1.

[0101] Convolution effective size Operation matrix size Consumption resource proportion Real-time processing speed and demand 16*16 32*32 0.60 9.45 48*16 64*32 0.60 7.56 48*48 64*64 0.60 5.04 112*112 128*128 0.69 18.5

[0102] Since the transmission data packets are split during horizontal filling, the horizontal expansion of the initial valid matrix can be omitted during expansion. Instead, during subsequent horizontal filling, the transmission data packets at the matrix edges are split to obtain the horizontal filling data and the horizontal expansion data. In one or more embodiments of this application, the process of avoiding edge effects can be divided into vertical expansion, obtaining the vertical filling data, the horizontal filling data, and the horizontal expansion data. After obtaining the initial valid matrix in step S101, this application performs vertical expansion on each initial valid matrix. Therefore, the process in step S101... Figure 3 In the example shown, the initial effective matrix adjacent to the upper edge is vertically expanded to obtain a target effective matrix of 48*56; the initial effective matrix adjacent to the lower edge is expanded to obtain a target effective matrix of 64*32; and the initial effective matrix located on the non-edge of the wafer image is expanded to obtain a target effective matrix of 64*48. That is, in this embodiment, the target effective matrix is ​​only vertically expanded. Then, while acquiring the read data, the padding data used to fill the target effective matrix at the wafer image edges, and the expansion data for horizontal expansion of each target effective matrix are also acquired, thereby avoiding edge effects in subsequent convolution preprocessing.

[0103] Since the wafer image is stored in memory as a row data stream, it is stored row by row in the memory, such as 9072*2048. Therefore, without horizontal scaling in this embodiment, this application can read the number of target valid matrices involved in a single row at once when acquiring the read data of the target valid matrix, such as 9072 / 48 = 189, to improve the efficiency of acquiring and reading data. However, due to memory resource limitations, there is a maximum read length for a single read (e.g., the maximum read length for DDR memory is 4KB). Therefore, in this application, the single read length can be set in advance based on memory resource limitations, the number of logical units, the required actual processing speed, etc.

[0104] like Figure 6 The diagram illustrates data read from memory according to an embodiment of this application. In this diagram, the size of a single wafer image is 9072*2048. This application can set the single read length to 71 data packets (2272 bytes), meaning 71 data packets (tdata1-tdata69) can be read from memory at a time. Therefore, one row of data in the wafer image requires 4 reads (9072 / 32≈284, 284 / 71=4 reads). The first target valid matrix involves data from rows 0 to 55 (56 rows in total) in the wafer image, and the second target valid matrix involves data from rows 40 to 103 (64 rows in total). This application can set a read flag signal (mem_rd_flag), which indicates whether the memory read state machine is enabled and positional operations are performed on the target valid matrix; a high-active flag indicates that the memory read state machine is active. The number of burst reads (rd_burst) in each read operation of the memory read state machine is equal to the matrix height of the target valid matrix.

[0105] In one or more embodiments of this application, the process of partitioning the initial effective matrix refers to using the effective convolution size as a block step to set the image traversal signal, and then partitioning the wafer image in the memory based on the image traversal signal to obtain multiple initial effective matrices, as detailed below:

[0106] First, this application determines the block stepping of the wafer image based on the effective convolution size, and sets the image traversal signal based on the block stepping. Second, based on the image traversal signal, this application controls the FPGA's memory to divide the wafer image in the memory according to the block stepping, obtaining multiple initial effective matrices.

[0107] It should be noted that the image traversal signals include the horizontal offset signal (tdata_coord_x_base), the image switching signal (image_coord_y_base), and the vertical block signal (matrix_coord_y). The definition of each image traversal signal is as follows:

[0108] tdata_coord_x_base: The position coordinates of the currently read transmission data packet (tdata) in its corresponding linescan (row in the wafer image). The initial value is 0, the step is rd_burst_len, and the calculation is performed after all the target valid matrices (matrix_pixel) involved in the linescan are read. After reading a linescan, the value is reset to zero before starting the next round of calculation.

[0109] image_coord_y_base: The position of the wafer image containing the currently read transmission data packet in the memory among all cached images; the initial value is 0, the step is the image height, and the calculation time is when the last set of target effective matrices of a wafer image is read; when all cached wafer images are read, the value is reset to zero, and the next round of calculation is started.

[0110] matrix_coord_y: The Y-axis coordinate (row direction) of the target valid matrix where the currently read transmission data packet is located in the wafer image; the initial value is 0, the step is 1, and the calculation time is when the last group of rd_burst of the last group of matrices of the line scan is read; when the last group of rd_burst of the last group of matrices at the bottom of a wafer image is read, it is reset to zero, and the next round of calculation is started.

[0111] Continuing with the previous example, assuming the wafer image size is 9072*2048, and the single read length rd_burst_len is 71 transmission data packets. The horizontal offset is calculated based on the size of 71 transmission data packets, and max_matrix_coord_y = image_height / 48. For example, 2048 / 48 rounded down results in 43 sets of matrices, each containing 9072 / 48 = 189 target effective matrices.

[0112] In one or more embodiments of this application, the process of expanding each initial valid matrix is ​​a process of adjusting the position identifier parameters corresponding to the initial valid matrix based on the image traversal signal, as detailed below:

[0113] First, for each initial valid matrix, this application determines the position identifier parameters of the edge rows in the initial valid matrix based on the image traversal signal. Second, for each initial valid matrix, the position identifier parameters of each edge row are adjusted according to the edge effect size. Finally, for each initial valid matrix, the target valid matrix after expansion is determined based on the adjusted position identifier parameters.

[0114] It should be noted that this application can set a first row position signal and a last row position signal to control the memory to acquire and read data according to the target valid matrix. The definitions of the first row position signal and the last row position signal are as follows:

[0115] First row position signal (image_coord_y_l): The Y-axis coordinate of the first row of the target valid matrix containing the currently read transmission data packet in all wafer images cached in memory. When reading each set of target valid matrices, additional extended data of size equal to the edge effect size above that set of target valid matrices needs to be read for vertical expansion; however, since there is no data for the wafer image above the first set of target valid matrices, vertical expansion is not possible. The first case is when matrix_coord_y = 0, in which case image_coord_y_l = image_coord_y_base; in other cases, image_coord_y_l = image_coord_y_base + image_coord_valid_y_l - matrix_pad_size, where matrix_pad_size is the edge effect size.

[0116] The last row position signal (image_coord_y_r) is the Y-axis coordinate of the last row of the target valid matrix containing the currently read transmission data packet within all wafer images cached in memory. When reading each set of target valid matrices, additional extended data of size equal to the edge effect size must be read below that set for vertical expansion; however, since there is no wafer image data below the last set of target valid matrices, vertical expansion is not possible. The first case is when the last set of target valid matrices is present, i.e., when matrix_coord_y = max_matrix_coord_y, then image_coord_y_r = image_coord_y_base + image_coord_valid_y_r; in other cases, image_coord_y_r = image_coord_y_base + image_coord_valid_y_r + matrix_pad_size.

[0117] Of course, in addition to the image traversal signal, first row position signal, and last row position signal mentioned above, this application also includes other matrix position calculation signals for controlling the read operation, so as to obtain the read data from the memory according to the matrix read form of the target valid matrix. The matrix position calculation signals are as follows:

[0118] Valid first row position signal (image_coord_valid_y_l): The position of the first row of the initial valid matrix (valid_matrix_pixel) where the currently read transmission data packet is located in the wafer image; the initial value is 0, the step is valid_matrix_size (e.g., the effective convolution size is 48 in the vertical direction), and the calculation timing is when the last group of rd_burst of the last group of matrices of the line scan is read (e.g., the last group of 4*71 transmission data packets); when the last group of rd_burst of the last group of matrices at the bottom of an image is read, it is reset to zero, and the next round of calculation is started.

[0119] Valid Last Row Position Signal (image_coord_valid_y_r): The position of the last row of the initial valid matrix containing the currently read transmission data packet within its wafer image; the initial value is valid_matrix_size, incremented by valid_matrix_size, and calculated when the last set of rd_burst of the last set of matrices in the line scan is read; if the read position is at max_matrix_coord_y, the value of this signal = image_height, and it returns to zero when the last set of rd_burst of the last set of matrices in max_matrix_coord_y is reached, and then the next round of calculation begins.

[0120] Initial address signal for the first row (image_coord_y_l_addr): The initial address of the linescan in memory where the first row of the target valid matrix containing the currently read transmission data packet is cached; the value of this initial address signal for the first row is image_coord_y_l * image_width, where image_width is the image width.

[0121] Last row address initial signal (image_coord_y_r_addr): The initial address of the last row of the target valid matrix containing the currently read transmission data packet, cached in memory by the linescan; the value of this last row address signal is image_coord_y_r * image_width.

[0122] First-row address actual signal (first_burst_inmatrix): The actual address in memory of the first row of the target valid matrix where the currently read transmission data packet is located; based on the above first-row address initial signal, the value of the first-row address actual signal can be obtained as image_coord_y_l_addr+tdata_coord_x_base.

[0123] The actual address of the last row (end_burst_inmatrix): the actual address of the last row of the target valid matrix where the currently read transmission data packet is located in DDR4; based on the above actual address of the last row, the value of the actual address of the last row can be obtained as image_coord_y_r_addr+tdata_coord_x_base.

[0124] Memory read address signal (ddr_araddr): The read address of the ddr; the initial value is 0, the step is the image width (image_width), and the calculation time is when a group of rd_burst (such as a group of 4*71 transmission data packets) is read; when ddr_araddr = end_burst_inmatrix, ddr_araddr remains unchanged.

[0125] Memory read status signal (mem_rd_flag): Enables the operation of the memory read state machine and the matrix position calculation signal mentioned above. It is active high. The number of rd_bursts read by the DDR read state machine each time is equal to the matrix height of the current actual target effective matrix.

[0126] Continuing with the previous example, based on the effective convolution size of 48 pixels, the block step for traversing the wafer image (9072*2048) is determined as 2048 / 48 = 42 remainder 32. Therefore, the vertical expansion block step is 48 and 32, meaning the initial effective matrix height for the first 42 rows is 48, and the initial effective matrix height for the last row is 32. That is, the initial effective matrix for the first row involves data from rows 0 to 47 of the wafer image, the initial effective matrix for the second row involves data from rows 48 to 97, and so on, deriving the position marker parameters for each row of the initial effective matrix according to the block step. Adjusting the position marker parameters of each initial effective matrix based on the edge effect size of 8 yields the expanded target effective matrix. Thus, the target effective matrix for the first row involves data from rows 0 to 55 of the wafer image, the target effective matrix for the second row involves data from rows 40 to 103, and so on.

[0127] In one or more embodiments of this application, the application can split the transmission data packets at the beginning and end of each row in the current read matrix based on the edge effect size to obtain the sub-data packet sequences corresponding to the first and last transmission data packets, respectively. If the current read matrix does not trigger horizontal line filling, the starting sub-data packet corresponding to the first transmission data packet and the ending sub-data packet corresponding to the last sub-data packet in the current read matrix are used as extended data for horizontally expanding the target valid matrix adjacent to the current valid matrix.

[0128] It should be noted that, based on the edge effect size, this application requires at least splitting the transmission data packets at the beginning and end of each row in the target effective matrix so that subsequent horizontal expansion / filling can be performed based on the sub-data packets split from the beginning and end. Using the example in step S100, with an edge effect size of 8 pixels and a wafer image acquisition format of Mono8 (8 bits = 1 pixel), the size of the starting and ending sub-data packets should be 8*8 = 64 bits.

[0129] To improve the processing speed of image processing by convolution preprocessing and graphics processing units, read data and padding data can be converted into pixel data. In one or more embodiments of this application, before step S104, this application can perform bit-width conversion on read data and padding data according to the quantization bit depth of pixels in the wafer image, thereby converting the acquired read data and padding data into pixel data. Since the splitting of transmission data packets is similar to the bit-width conversion process, this application can first split each transmission data packet once, and then perform a second split based on the quantization bit depth. In one or more embodiments of this application, firstly, before step S104, this application can split each transmission data packet of the current read matrix according to the edge effect size to obtain a sequence of sub-data packets of the transmission data packet, where the size of each sub-data packet in the sub-data packet sequence is the edge effect size. Secondly, this application can perform bit-width conversion on the data after horizontal and vertical padding of each target effective matrix based on the quantization bit depth of the wafer image to obtain first data, and then perform convolution preprocessing on the first data to obtain target data.

[0130] Continuing with the previous example, the wafer image acquisition format is Mono8 (8 bits = 1 pixel), meaning the quantization bit depth of each pixel in the wafer image is 8 bits; the edge effect size is 8 pixels; and the size of the transmission data packet is 256 bits. This application performs a first split on the transmission data packet (256 bits) based on the edge effect size (8 bits * 8 pixels = 64 bits), achieving a bit width conversion from 256 bits to 64 bits. Then, a second split is performed based on the quantization bit depth (8 bits), achieving a bit width conversion from 64 bits to 8 bits.

[0131] like Figure 7 The diagram illustrates horizontal padding based on split transmission data packets according to an embodiment of this application. Following the previous example, in this diagram, based on the edge effect size (8 pixels * 8 bits = 64 bits), this application splits the transmission data packet (256 bits = 32 pixels) into a sequence of sub-data packets (rdata_0, rdata_1, rdata_2, rdata_3), each sub-data packet being 64 bits in size. The starting sub-data packet at the beginning of the sequence is left_pad_rdata; the ending sub-data packet at the end of the sequence is right_pad_rdata. The starting and ending sub-data packets are used for horizontal padding. Since the last transmission data packet in each row is only partially valid, the ending sub-data packet is the second sub-data packet of that last transmission data packet. Because the target valid matrix is ​​48 * 64, meaning it includes 48 pixels horizontally, and a transmission data packet is 32 pixels, the target valid matrix needs to receive two transmission data packets. rdata[255:0] represents the real-time reading of a 256-bit transmission data packet, while rdata_X_dY represents the data after a delay of Y periods for the Xth data segment, used for timing alignment, so as to complete the subsequent 8+48+8=64 horizontal expansion and / or horizontal padding. This diagram is only used as a partial illustration of timing alignment. data0 represents the first byte (8 bits), which is rdata_0[7:0], data1 represents the second byte (8 bits), which is rdata_0[15:8], and data2, data3, and so on. matrix_pixel_N represents the target effective matrix after the Nth horizontal padding and horizontal expansion, that is, the last obtained matrix_pixel data is stored in BRAM according to the matrix.

[0132] This application provides an FPGA-based image processing method. Accordingly, it also provides a specific embodiment of an FPGA-based image processing system.

[0133] like Figure 8 As shown in the schematic diagram, an embodiment of this application provides a structural diagram of an FPGA-based image processing system. The system 800 includes a memory buffer unit 801, a pixel positioning unit 802, and a convolution preprocessing unit 803. The memory buffer unit includes a write state machine 804, a read state machine 805, a control subunit 806, and a matrix position subunit 807. The pixel positioning unit includes a first-in-first-out buffer subunit 808, a fill judgment subunit 809, a horizontal fill subunit 810, and a vertical fill subunit 811. The specific functions of each unit are as follows:

[0134] First, the main function of the memory buffer unit (Buffer, such as DDR4_Buffer) in this application is to implement the caching of wafer images and to locate the position of matrix_pixel to obtain the tdata corresponding to the read data.

[0135] In some embodiments, the above-mentioned write state machine (awr_state) is used to cache the received tdata into DDR4, and to perform sequential caching according to the receiving order. The single write length (wr_burst_len) can be set according to actual needs. For example, wr_burst_len = 128 transmission data packets. If a transmission data packet is 256 bits, then the single write length is about 4KB.

[0136] In some embodiments, the aforementioned read state machine (ard_state) is used to read the matrix_pixel position calculated according to the matrix_position_cal. The rd_burst_len is set based on the number of logic units in the FPGA, the required processing speed, and the image width. Continuing with the previous example, if rd_burst_len = 71, then completing one linescan read requires four rd_burst reads.

[0137] In some embodiments, the matrix position sub-unit (matrix_position_cal) is used to calculate signals based on the matrix positions to obtain the tdata of the matrix_pixel, so as to obtain the read data.

[0138] Secondly, the main function of the pixel positioning unit (matrix_pixel_get) in this application is to output tdata[255:0] from the received memory buffer unit and convert it into pixel_data[7:0] for each matrix_pixel, so as to perform convolution preprocessing and image processing later.

[0139] In some embodiments, the above-mentioned first-in-first-out (FIFO_OUT) buffer subunit is used for first-in-first-out (FIFO_OUT) buffering. Its function is to ensure that the data fifo_out_data read from the subunit is continuous, and each read meets the length rd_burst_len, so as to avoid the situation where the read data may be non-continuous when the memory (such as DDR4) is reading.

[0140] In some embodiments, the aforementioned filling determination subunit (matrix_pos_cal) is used to calculate the position of the matrix_pixel containing the data currently being processed in the wafer image.

[0141] It should be noted that the padding determination subunit is divided into a vertical padding determination subunit and a horizontal padding determination subunit. The vertical padding determination subunit (matrix_pos_cal_ramout) calculates the position information required for the vertical padding subunit to determine which matix_pixel the data being processed belongs to and whether vertical padding is required. The horizontal padding determination subunit (matrix_pos_cal_ramin) calculates the position information required for the horizontal padding subunit to determine which matix_pixel the transmitted data packet being processed belongs to and whether horizontal padding is required.

[0142] Continuing with the previous example, the target effective matrix includes three matrix heights: the matrix height for rows 0-55 is 56, the matrix height for rows 40-103 is 64, and so on. Therefore, the values ​​for current_matrix_ysize can be 55, 63, and 39.

[0143] In some embodiments, the aforementioned horizontal padding subunit is used to first split the transmission data packet, perform horizontal padding, divide it according to matrix_pixel, and then store the obtained matrix_pixel_data in matrix form.

[0144] In some embodiments, the vertical fill subunit described above is used to perform bit width conversion and vertical fill.

[0145] It should be noted that the vertical fill sub-unit determines whether the target effective matrix needs to be vertically filled based on the position (current_matrix_ysize) of the target effective matrix in the wafer image calculated by the fill judgment sub-unit, and then performs the read address addrb operation to obtain the vertical fill data and the read data of the target effective matrix.

[0146] Finally, the main function of the convolution preprocessing unit in this application is to perform convolution preprocessing on the read data and padding data to obtain the target data, and then send the target data to the graphics processing unit that performs image processing.

[0147] like Figure 9aThe diagram shows a flowchart of vertical padding provided in an embodiment of this application. In this diagram, the padding determination subunit 809 can: S900: determine whether to trigger top vertical padding (top_pad) 900 on the wafer image based on whether the position of the target effective matrix in the wafer image is zero. Simultaneously, S901: determine whether to trigger bottom vertical padding (bottom_pad) 901 on the wafer image based on whether the position of the target effective matrix in the wafer image is max_matrix_coord_y (refer to the relevant content of the matrix position calculation signal mentioned above). If top and bottom vertical padding are not triggered, the case is normal 902.

[0148] Furthermore, the pixel data after bit-width conversion is stored in the BRAM cache. To read the pixel data, a read enable signal and a read address signal (addrb) are required. The wafer image lines are counted starting from 0. For example... Figure 9b The diagram shown is a schematic representation of the bottom vertical filling process according to an embodiment of this application. The data acquisition process for implementing bottom vertical filling includes steps S902 to S911, as follows:

[0149] S902: Initialize the values ​​of x_cnt, y_cnt, addrb_e, and addrb_m; S903: Enable memory read; S904: (After enabling memory read) Increment x_cnt; S905: Check if x_cnt is 63 (x_cnt = 63); S906: If x_cnt = 63, increment y_cnt; S907: Check if y_cnt is greater than 39 (y_cnt > 39); S908: If y_cnt > 39, then addrb = addrb_e; S909: If y_cnt is not greater than 39, then addrb = addrb_m, and increment addrb_m; S910: If x_cnt = 63, decrement addrb_e by 127; S911: If x_cnt is not equal to 63, increment addrb_e.

[0150] like Figure 9c The diagram shown is a schematic representation of the top vertical fill process according to an embodiment of this application. The data acquisition process for implementing the top vertical fill includes steps S912 to S921, as follows:

[0151] S912: Initialize the values ​​of x_cnt, y_cnt, addrb_t, and addrb_m; S913: Enable memory read; S914: (After enabling memory read) Increment x_cnt; S915: Check if x_cnt is 63 (x_cnt = 63); S916: If x_cnt = 63, increment y_cnt; S917: Check if y_cnt is less than 8 (y_cnt < 8); S918: If y_cnt < 8, then addrb = addrb_t; S919: If y_cnt is not less than 8, then addrb = addrb_m, and increment addrb_m; S920: If x_cnt = 63, decrement addrb_t by 127; S921: If x_cnt is not equal to 63, increment addrb_t.

[0152] like Figure 9d The diagram illustrates the process of obtaining pixel data when vertical fill is not triggered, as provided in an embodiment of this application. In this diagram, the case where top and bottom vertical fill is not triggered is the general case 902. The data acquisition process for the general case includes steps S922 to S924, as follows:

[0153] S922: Initialize addrb to zero (addrb = 0); S923: Enable memory read; S924: With memory read enabled, increment addrb by one.

[0154] In Figure 9a , Figure 9b , Figure 9c as well as Figure 9d The definitions and functions of the above parameters are as follows:

[0155] x_cnt: The x-axis counter of the matrix, with a counting period of 0-63. At least 6 bits are required for x-axis counting, i.e., x_cnt[5:0]. When the read enable signal is valid, x_cnt is accumulated during the clock cycle, and after accumulating to 63, it is reset to zero, and this action is repeated. y_cnt: The y-axis counter of the matrix, with a counting period of 0-63. At least 6 bits are required for x-axis counting, i.e., y_cnt[5:0]. When the read enable signal is valid and x_cnt = 63, it is accumulated during the clock cycle. addrb: The read address signal of the BRAM. When in the bottom vertical fill 901 stage, if y_cnt > 39, addrb = addrb_e, otherwise addrb = addrb_m; in other cases, addrb = addrb_m.

[0156] If in the top vertical fill phase and y_cnt > 7, addrb_m is incremented by one; if not in the top vertical fill phase, addrb_m is incremented by one. addrb_e: The read address for the bottom vertical fill, initially 2496 (the first pixel of row 39); if the current action is in the bottom vertical fill phase, when y_cnt > 39, addrb_e is activated: when x_cnt = 63 (when a set of valid target matrices has been read), addrb_e is decremented by 127, i.e., from the address of the last pixel in row n to the address of the first pixel in row (n-1); if x_cnt is not equal to 63, addrb_e is incremented by 1. In the top vertical fill phase, if y_cnt < 8, addrb = addrb_t; otherwise, addrb = addrb_m. `addrb_t`: The read address for top vertical fill, initially set to 448 (first pixel of row 7). If in the top vertical fill phase, `addrb_t` is enabled when `y_cnt` < 8 (0-7). When `x_cnt` = 63 (after reading a set of valid target matrices), `addrb_t` is decremented by 127, i.e., from the address of the last pixel in row n to the address of the first pixel in row (n-1). If `x_cnt` is not equal to 63, `addrb_t` is incremented by 1. `addrb_m`: The read address without fill, initially set to 0 (first pixel of row 0).

[0157] This application provides an image processing method based on FPGA. Accordingly, it also provides a specific embodiment of an FPGA-based image processing apparatus.

[0158] like Figure 10 As shown in the figure, an FPGA-based image processing device 1000 provided in this application embodiment includes a matrix positioning module 1001, an image traversal module 1002, a memory reading module 1003, and a convolution preprocessing module 1004.

[0159] The matrix positioning module 1001 is used to divide the wafer image in the memory of the FPGA into multiple initial effective matrices according to a preset convolution effective size, and to expand each initial effective matrix according to a preset edge effect size to obtain a target effective matrix corresponding to each initial effective matrix.

[0160] The image traversal module 1002 is used to determine the target read range of each target effective matrix in the memory based on the matrix coordinates of the target effective matrix in the image coordinate system of the wafer image and the matrix size of the target effective matrix.

[0161] The memory read module 1003 is used to obtain read data of each target effective matrix from the memory according to the target read range, and in the process of obtaining the read data, obtain fill data for filling the target effective matrix located at the edge of the wafer image according to the edge effect size;

[0162] The convolution preprocessing module 1004 is used to perform convolution preprocessing on the read data and the padding data according to the pixel data groups corresponding to each of the target effective matrices to obtain target data, and send the target data to the graphics processing unit that performs image processing.

[0163] In some embodiments, the image traversal module 1002 is specifically configured to: determine the pixel row coordinates of the first and last rows of the target effective matrix in the image coordinate system based on the matrix row coordinates and the matrix height of the target effective matrix; determine the read start address based on the pixel row coordinates corresponding to the first row of the target effective matrix and the starting base address of the wafer image in the memory; determine the read end address based on the pixel row coordinates corresponding to the last row of the target effective matrix, the starting base address, and the matrix width of the target effective matrix; and determine the target read range of the target effective matrix in the memory based on the read start address and the read end address.

[0164] In some embodiments, the memory reading module 1003 is specifically configured to: before dividing the wafer image in the FPGA's memory into multiple initial effective matrices according to a preset convolution effective size, transmit and write the row data stream of the wafer image after scanning the wafer into the FPGA's memory row by row, wherein each row of the row data stream includes multiple transmission data packets; take the target effective matrix corresponding to the currently read transmission data packet as the current reading matrix; determine the target column coordinates of the first or last column in the current reading matrix in the image coordinate system, and if the target column coordinates are the same as the start or end column coordinates in the wafer image, obtain the target effective matrix for use in the image coordinate system. The horizontal filling data includes determining, based on the edge effect size, a horizontal filling region adjacent to the first or last column of the current read matrix in the wafer image, and filling the horizontal filling region; determining the target row coordinates of the first or last row of the current read matrix in the image coordinate system, and if the target row coordinates are the same as the start or end row coordinates in the wafer image, obtaining the vertical filling data, which includes determining, based on the edge effect size, a vertical filling region adjacent to the first or last row of the current read matrix in the wafer image, and filling the vertical filling region.

[0165] In some embodiments, the memory read module 1003 described above can also be used to: split the transmission data packets located at the beginning and end of each row in the current read matrix based on the edge effect size to obtain a sequence of sub-data packets of the transmission data packets, wherein the sub-data packet sequence includes at least a starting sub-data packet at the beginning of the sequence and a terminating sub-data packet at the end of the sequence, and the size of the starting sub-data packet and the terminating sub-data packet is the edge effect size; when the target column coordinate is the same as the starting column coordinate, copy the starting sub-data packet as filling data for mirroring filling in the horizontal filling area; when the target column coordinate is the same as the terminating column coordinate, copy the terminating sub-data packet as filling data for mirroring filling in the horizontal filling area.

[0166] In some embodiments, the memory read module 1003 described above can also be used to: when the target row coordinates are the same as the starting row coordinates, determine the starting copy range of the current read matrix according to the edge effect size, and copy the data within the starting copy range as fill data for mirror filling in the vertical fill area; when the target row coordinates are the same as the ending row coordinates, determine the ending copy range according to the edge effect size and the matrix height of the current read matrix, and copy the data within the ending copy range as fill data for mirror filling in the vertical fill area.

[0167] In some embodiments, the memory read module 1003 described above can also be used to: store the read data and the fill data into a first-in-first-out (FIFO) cache unit; and obtain the read data and the fill data from the FIFO cache unit based on the size of the operation matrix after expansion and / or filling of each initial valid matrix.

[0168] In some embodiments, the matrix positioning module 1001 is specifically used to: determine the block stepping for traversing the wafer image according to the effective convolution size, and set an image traversal signal based on the block stepping; and control the memory to divide the wafer image in the memory according to the block stepping based on the image traversal signal to obtain a plurality of the initial effective matrices.

[0169] In some embodiments, the matrix positioning module 1001 described above can also be used to: for each initial effective matrix, determine the position identifier parameters of the edge rows in the initial effective matrix according to the image traversal signal; for each initial effective matrix, adjust the position identifier parameters of each edge row according to the edge effect size; and for each initial effective matrix, determine the target effective matrix after the initial effective matrix is ​​expanded based on the adjusted position identifier parameters.

[0170] This application provides an image processing method based on FPGA. Accordingly, it also provides a specific embodiment of an FPGA-based image processing device.

[0171] Figure 11 A schematic diagram of the hardware structure of an FPGA-based image processing device provided in an embodiment of this application is shown.

[0172] The processing device for wafer morphology scanning signals may include a processor 1101 and a memory 1102 storing computer program instructions.

[0173] Specifically, the processor 1001 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0174] Memory 1102 may include mass storage for data or instructions. For example, and not limitingly, memory 1102 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1102 may include removable or non-removable (or fixed) media. Where appropriate, memory 1102 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1102 is non-volatile solid-state memory.

[0175] The processor 1101 implements any of the FPGA-based image processing methods described above by reading and executing computer program instructions stored in the memory 1002.

[0176] In one example, the electronic device may also include a communication interface 1103 and a bus 1110. Wherein, as... Figure 11As shown, the processor 1101, memory 1102, and communication interface 1103 are connected through bus 1110 and complete communication with each other.

[0177] The communication interface 1103 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0178] Bus 1110 includes hardware, software, or both, that couples the components of the electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1110 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0179] Furthermore, in conjunction with the FPGA-based image processing method described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the FPGA-based image processing methods described in the above embodiments.

[0180] In addition, in conjunction with the FPGA-based image processing method in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device performs an FPGA-based image processing method as provided in any aspect of the above embodiments of this application.

[0181] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0182] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0183] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0184] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0185] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. An image processing method based on FPGA, characterized in that, The method is applied to FPGA and includes: According to the preset effective convolution size, the wafer image in the FPGA memory is divided into multiple initial effective matrices, and each initial effective matrix is ​​expanded according to the preset edge effect size to obtain the target effective matrix corresponding to each initial effective matrix; For each target effective matrix, the target read range of the target effective matrix in the memory is determined based on the matrix coordinates of the target effective matrix in the image coordinate system of the wafer image and the matrix size of the target effective matrix; Read data for each target effective matrix is ​​obtained from the memory according to the target read range, and during the process of obtaining the read data, fill data for filling the target effective matrix located at the edge of the wafer image is obtained according to the edge effect size; The read data and the filling data are preprocessed by convolution according to the pixel data groups corresponding to each target effective matrix to obtain target data, and the target data is sent to the graphics processing unit that performs image processing.

2. The method as described in claim 1, characterized in that, Based on the matrix coordinates of the target effective matrix in the image coordinate system of the wafer image and the matrix size of the target effective matrix, the target read range of the target effective matrix in the memory is determined, specifically including: Based on the row coordinates of the target effective matrix in the image coordinate system and the matrix height of the target effective matrix, determine the pixel row coordinates of the first and last rows of the target effective matrix in the image coordinate system, respectively. The read start address is determined based on the pixel row coordinates corresponding to the first row in the target effective matrix and the starting base address of the wafer image in the memory. The end address of the read is determined based on the pixel row coordinates corresponding to the last row in the target effective matrix, the starting base address, and the matrix width of the target effective matrix; Based on the read start address and the read end address, the target read range of the target valid matrix in the memory is determined.

3. The method as described in claim 1, characterized in that, Before dividing the wafer image in the FPGA's memory into multiple initial effective matrices according to a preset effective convolution size, the method further includes: The row data stream of the wafer image after scanning the wafer is written into the memory of the FPGA by transmitting it row by row, wherein each row of the row data stream includes multiple transmission data packets; During the acquisition of the read data, based on the edge effect size, fill data is obtained for filling the target effective matrix located at the edge of the wafer image, specifically including: The target valid matrix corresponding to the currently read transmission data packet is used as the current read matrix; Determine the target column coordinates of the first or last column in the current read matrix in the image coordinate system. If the target column coordinates are the same as the start or end column coordinates in the wafer image, obtain the filling data for horizontal filling. The horizontal filling includes determining a horizontal filling region adjacent to the first or last column in the current read matrix in the wafer image according to the edge effect size, and filling the horizontal filling region. Determine the target row coordinates of the first or last row in the current read matrix in the image coordinate system. If the target row coordinates are the same as the starting or ending row coordinates in the wafer image, obtain the filling data for vertical filling. The vertical filling includes determining a vertical filling region adjacent to the first or last row in the current read matrix in the wafer image based on the edge effect size, and filling the vertical filling region.

4. The method as described in claim 3, characterized in that, Obtaining the fill data for horizontal filling specifically includes: Based on the edge effect size, the transmission data packets located at the beginning and end of each row in the current reading matrix are split to obtain a sequence of sub-data packets of the transmission data packets. The sequence of sub-data packets includes at least a starting sub-data packet at the beginning of the sequence and a terminating sub-data packet at the end of the sequence, and the size of the starting sub-data packet and the terminating sub-data packet is the edge effect size. If the target column coordinates are the same as the starting column coordinates, the starting sub-data packet is copied as fill data for mirroring and filling the horizontal fill area; If the target column coordinates are the same as the termination column coordinates, the termination sub-data packet is copied as fill data for mirroring the filling of the horizontal fill area.

5. The method as described in claim 3, characterized in that, After obtaining the fill data for vertical filling, the specific steps include: If the target row coordinates are the same as the starting row coordinates, the starting copy range of the current read matrix is ​​determined according to the edge effect size, and the data within the starting copy range is copied as fill data for mirroring the filling of the vertical fill area; If the target row coordinates are the same as the termination row coordinates, the termination copy range is determined based on the edge effect size and the matrix height of the current read matrix, and the data within the termination copy range is copied as fill data for mirroring the vertical fill area.

6. The method as described in claim 1, characterized in that, Before performing convolution preprocessing on the read data and the padding data according to the pixel data groups corresponding to each of the target effective matrices, the method further includes: The read data and the fill data are stored in a first-in-first-out cache unit; Based on the size of the operational matrix after expansion and / or padding of each initial valid matrix, the read data and the padding data are obtained from the first-in-first-out cache unit.

7. The method as described in claim 1, characterized in that, Based on a preset effective convolution size, the wafer image in the memory is divided into multiple initial effective matrices, specifically including: Based on the effective size of the convolution, determine the block stepping for traversing the wafer image, and set the image traversal signal based on the block stepping; Based on the image traversal signal, the memory is controlled to divide the wafer image in the memory according to the block stepping, so as to obtain multiple initial effective matrices.

8. The method as described in claim 7, characterized in that, Based on a preset edge effect size, the range of each initial effective matrix in the wafer image is expanded to obtain the target effective matrix corresponding to each initial effective matrix, specifically including: For each of the initial valid matrices, the position identifier parameters of the edge rows in the initial valid matrix are determined based on the image traversal signal; For each of the initial valid matrices, the position identifier parameters of each edge row are adjusted according to the edge effect size; For each of the initial valid matrices, the target valid matrix after expansion is determined based on the adjusted position identifier parameters.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement an FPGA-based image processing method as described in any one of claims 1-8.

10. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs an FPGA-based image processing method as described in any one of claims 1-8.

Images