Differential dynamic microscopy image processing method, device and system based on FPGA

Abstract

The present invention relates to a differential dynamic microscopic image processing method based on FPGA, which comprises configuring a first write buffer area and a first read buffer area in a memory inside the FPGA chip, and configuring a first address space in a first memory outside the FPGA chip; acquiring microscopic image information and converting it into a grayscale image data stream; storing a first frame of grayscale image data in the grayscale image data stream in a first write buffer area, and writing the first frame of grayscale image data in the first write buffer area into a first address space; when receiving the first pixel data of a new frame of grayscale image data, reading out the first frame of grayscale image data in the first address space and storing it in a first read buffer area; reading the first frame of grayscale image data from the first read buffer area, and performing inter-frame difference calculation with the new frame of grayscale image data to obtain differential image data at a new frame moment; performing a two-dimensional Fourier transform on the differential image data to obtain Fourier spectrum image data, which is the result image information.

Description

Technical Field

[0001] The present invention relates to the technical field of particle microscopic image processing, and in particular to a differential dynamic microscopic image processing method, device and system based on FPGA. Background Art

[0002] In recent years, my country's nano-related technologies have developed rapidly, and the research on the properties of nanoparticles has also attracted more and more researchers. Among them, the particle size and motion characteristics of nanoparticles are of great significance to improving the understanding of the properties of nanoparticles. In recent years, researchers have combined the basic principles of light scattering technology and microscopy technology to propose a new tool for studying nanoparticles and other substances—digital Fourier microscopy. This technology obtains the microscopic image of the sample in real space and performs Fourier analysis on the image in reciprocal space to obtain the information of the sample. Differential dynamic microscopy (DDM) technology is a type of digital Fourier microscopy technology. It reconstructs the dynamic scattering pattern through the changes in the light intensity of the microscopic image in the time domain and space domain, thereby obtaining the dynamic information of the sample. However, this method involves the processing of scattered light images with the characteristics of large image data volume and high real-time performance. Although the existing CPU-based image processing system is highly versatile and easy to develop, it has the problems of large size, slow computing speed and high cost; the GPU-based image processing system has the disadvantage of high power consumption; the DSP-based image processing system has limited processing power and is difficult to complete complex algorithms.

[0003] FPGA chips have the characteristics of high parallelism and rich internal logic resources, but how to use FPGA chips to improve the processing speed of differential dynamic microscopy is still a technical problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a differential dynamic microscopy image processing method based on FPGA, which can utilize FPGA to implement image processing of differential dynamic microscopy technology.

[0005] The present invention is implemented by the following technical solution: a differential dynamic microscopic image processing method based on FPGA, comprising the steps of:

[0006] A first write cache area and a first read cache area are configured in a memory within the FPGA chip, and a first address space is configured in a first memory outside the FPGA chip;

[0007] Acquire microscopic image information, and convert the microscopic image information into a grayscale image data stream;

[0008] storing the first frame of the grayscale image data in the grayscale image data stream in the first write buffer area, and writing the first frame of the grayscale image data in the first write buffer area into the first address space;

[0009] When receiving the first pixel data of a new frame of the grayscale image data, reading out the first frame of grayscale image data in the first address space and storing it in the first read buffer area;

[0010] Reading the first frame of grayscale image data from the first read buffer area, and performing inter-frame difference calculation with the new frame of grayscale image data to obtain differential image data at a new frame moment;

[0011] Perform a two-dimensional Fourier transform on the differential image data to obtain corresponding Fourier spectrum image data, and determine the Fourier spectrum image data as result image information.

[0012] Compared with the prior art, the present invention provides a differential dynamic microscopy image processing method based on FPGA, which reasonably and efficiently utilizes the storage resources inside and outside the FPGA chip, can meet the processing requirements of large amounts of data, and can meet the requirements of high real-time performance, accelerate the calculation speed of the core algorithm module, reduce the system size, and has strong parallel capability.

[0013] Furthermore, writing the first frame of grayscale image data in the first write buffer area into the first address space includes the following steps: when the amount of the grayscale image data in the first write buffer area reaches a preset burst read / write length, burst writing the grayscale image data of the burst read / write length in the first write buffer area into the first address space until all the grayscale image data of the first frame are written into the first address space.

[0014] Further, the steps include:

[0015] A second write cache area, a second read cache area, an address cache area and a frame ID cache area are configured in the memory inside the FPGA chip, and a plurality of image cache spaces are configured in the second memory outside the FPGA chip;

[0016] Performing a two-dimensional Fourier transform on the differential image data to obtain corresponding Fourier spectrum image data comprises the steps of:

[0017] Performing a one-dimensional Fourier transform on the differential image data in a column direction or a row direction to obtain intermediate result matrix data;

[0018] Generate a frame ID according to the frame time of the intermediate result matrix data and store it in the frame ID buffer area, generate an intermediate buffer address according to the frame time of the intermediate result matrix data, store the intermediate result matrix data and the corresponding intermediate buffer address in the second write buffer area, and write the intermediate result matrix data in the second write buffer area into the corresponding image buffer space according to the intermediate buffer address in the second write buffer area;

[0019] Generate a corresponding intermediate buffer address according to the frame ID in the frame ID buffer, store the intermediate buffer address in the address buffer, read out the corresponding intermediate result matrix data in the image buffer space according to the intermediate buffer address in the address buffer, and store the intermediate result matrix data in the second read buffer;

[0020] Performing a one-dimensional Fourier transform in another direction on the intermediate result matrix data in the second read buffer area to obtain two-dimensional transformation result data;

[0021] The square of the amplitude of the absolute value of the two-dimensional transformation result data is calculated and determined as Fourier spectrum image data.

[0022] Furthermore, while storing the intermediate result matrix data and the corresponding intermediate cache address in the second write cache area, the method further includes the steps of: counting the intermediate result matrix or the intermediate cache address stored in the second write cache area, and when the count reaches a preset value, starting to write the intermediate result matrix data in the second write cache area into the corresponding image cache space according to the intermediate cache address in the second write cache area;

[0023] While writing the intermediate result matrix data in the second write cache area into the corresponding image cache space, it also includes the step of: counting the intermediate result matrix data written into the image cache space, and when the count reaches a preset value, stopping writing the intermediate result matrix data in the second write cache area into the corresponding image cache space.

[0024] Furthermore, while storing the intermediate cache address in the address cache area, the method further includes the steps of: counting the intermediate cache addresses stored in the address cache area, and when the count exceeds a preset value and the storage space of the second read cache area is not full, starting to read out the intermediate result matrix data in the corresponding image cache space according to the intermediate cache address in the address cache area;

[0025] While reading out the intermediate result matrix data in the corresponding image cache space and storing it in the second read cache area, it also includes the steps of: counting the intermediate result matrix data read out from the corresponding image cache space, and when the count reaches a preset value, stopping reading out the intermediate result matrix data in the corresponding image cache space; counting the intermediate result matrix or the intermediate cache address stored in the second write cache area, and if the count reaches a preset value, stopping reading out the intermediate result matrix data in the corresponding image cache space.

[0026] Based on the same inventive concept, the present invention also provides a differential dynamic microscopic image processing device based on FPGA, comprising:

[0027] A storage space configuration unit, configured to configure a first write cache area and a first read cache area in the memory within the FPGA chip, and to configure a first address space in a first memory outside the FPGA chip;

[0028] An image conversion unit, used for acquiring microscopic image information and converting the microscopic image information into a grayscale image data stream;

[0029] The inter-frame difference unit comprises a first write control module, a first read control module and a difference calculation module, wherein the first write control module is used to store the first frame of the grayscale image data in the grayscale image data stream in the first write buffer area, and write the first frame of the grayscale image data in the first write buffer area into the first address space; the first read control module is used to read out the first frame of the grayscale image data in the first address space and store it in the first read buffer area when receiving the first pixel data of a new frame of the grayscale image data; the difference calculation module is used to read the first frame of the grayscale image data from the first read buffer area, and perform inter-frame difference calculation with the new frame of the grayscale image data to obtain the difference image data at the time of the new frame;

[0030] The two-dimensional Fourier transform unit is used to perform a two-dimensional Fourier transform on the differential image data to obtain corresponding Fourier spectrum image data, and determine the Fourier spectrum image data as result image information.

[0031] Furthermore, the first write control module writes the first frame of grayscale image data in the first write buffer area into the first address space, including the steps of: when the data amount of the grayscale image data in the first write buffer area reaches a preset burst read / write length, burst writing the grayscale image data of the burst read / write length in the first write buffer area into the first address space until all the grayscale image data of the first frame are written into the first address space.

[0032] Further, the storage space configuration unit is used to configure a second write cache area, a second read cache area, an address cache area and a frame ID cache area in the memory within the FPGA chip, and configure multiple image cache spaces in the second memory outside the FPGA chip;

[0033] The two-dimensional Fourier transform unit includes a first transform module, a second write control module, a second read control module, a second transform module and a spectrum image calculation module. The first transform module is used to perform a one-dimensional Fourier transform in the column direction or the row direction on the differential image data to obtain intermediate result matrix data; the second write control module is used to generate a frame ID according to the frame time of the intermediate result matrix data and store it in the frame ID buffer area, generate an intermediate buffer address according to the frame time of the intermediate result matrix data, store the intermediate result matrix data and the corresponding intermediate buffer address in the second write buffer area, and read the intermediate result matrix data in the second write buffer area according to the intermediate buffer address in the second write buffer area. The matrix data is written into the corresponding image cache space; the second read control module is used to generate a corresponding intermediate cache address according to the frame ID in the frame ID cache area, store the intermediate cache address in the address cache area, read out the intermediate result matrix data in the corresponding image cache space according to the intermediate cache address in the address cache area and store it in the second read cache area; the second transformation module is used to perform a one-dimensional Fourier transform in another direction on the intermediate result matrix data in the second read cache area to obtain two-dimensional transformation result data; the spectrum image calculation module is used to calculate the square of the amplitude of the absolute value of the two-dimensional transformation result data, and determine it as Fourier spectrum image data.

[0034] Furthermore, the second write control module stores the intermediate result matrix data and the corresponding intermediate cache address in the second write cache area, and counts the intermediate result matrix or the intermediate cache address stored in the second write cache area. When the count reaches a preset value, the intermediate result matrix data in the second write cache area starts to be written into the corresponding image cache space according to the intermediate cache address in the second write cache area.

[0035] The second write control module writes the intermediate result matrix data in the second write cache area into the corresponding image cache space, and counts the intermediate result matrix data written into the image cache space. When the count reaches a preset value, the intermediate result matrix data in the second write cache area is stopped from being written into the corresponding image cache space.

[0036] The second read control module stores the intermediate cache address in the address cache area and counts the intermediate cache address stored in the address cache area. When the count exceeds a preset value and the storage space of the second read cache area is not full, the intermediate result matrix data in the image cache space corresponding to the intermediate cache address in the address cache area is read out.

[0037] The second read control module reads out the intermediate result matrix data in the corresponding image cache space and stores it in the second read cache area, while counting the intermediate result matrix data read out in the corresponding image cache space. When the count reaches a preset value, the reading of the intermediate result matrix data in the corresponding image cache space is stopped; the intermediate result matrix or the intermediate cache address stored in the second write cache area is counted. If the count reaches a preset value, the reading of the intermediate result matrix data in the corresponding image cache space is stopped.

[0038] Based on the same inventive concept, the present invention also provides an FPGA-based differential dynamic microscopy image processing system, comprising a microscope, a camera and an image processing unit, wherein the camera collects microscopic images in the microscope and transmits them to the image processing unit, and is characterized in that the image processing unit comprises an FPGA controller, a first memory and a second memory, the FPGA controller comprises a memory for storing computer programs and cache data and a processor for executing the computer program, and when the computer program is executed by the processor, the steps of the above-mentioned FPGA-based differential dynamic microscopy image processing method are implemented.

[0039] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the structure of the differential dynamic microscopy image processing system based on FPGA in the embodiment;

[0041] Figure 2 Schematic diagram of the process of the differential dynamic microscopy image processing method based on FPGA in the embodiment;

[0042] Figure 3 for Figure 2 The schematic diagram of the process of converting the microscopic image information into a grayscale image data stream in step S2 shown;

[0043] Figure 4 for Figure 2 The schematic diagram of the process of performing inter-frame difference processing on the grayscale image data stream in step S3 shown;

[0044] Figure 5 for Figure 2 The schematic diagram of the process of performing two-dimensional Fourier transform on the differential image data stream in step S4 shown;

[0045] Figure 6 is an exemplary state machine flow chart of the read and write control of the second memory 153 in a specific implementation;

[0046] Figure 7 is an exemplary state machine flow chart of intermediate cache address generation control in a specific implementation;

[0047] Figure 8 for Figure 2 The schematic diagram of the process of outputting the result image information in step S5 shown;

[0048] Fig. 9 Schematic diagram of the structure of the FPGA-based differential dynamic microscopy image processing device in the embodiment. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0050] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0052] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0053] The present invention improves the data storage and processing capabilities of FPGA for differential dynamic microscopic image processing by flexibly utilizing the resources within the FPGA chip and utilizing the storage resources outside the FPGA chip.

[0054] See also Figure 1 , which is a schematic diagram of the structure of the differential dynamic microscopy image processing system based on FPGA of this embodiment, the system includes a sample pool 11, a light source 12, a microscope 13, a camera 14, an image processing unit 15 and a display 16, wherein the sample pool 11 is used to hold a sample solution to be observed, such as a nanoparticle solution; the light of the light source 12 is directed to the sample pool 11 to provide illumination for the sample solution in the sample pool 11, and the light source 12 can be a white light source; the objective lens of the microscope 13 is aligned with the sample pool 11 to illuminate the sample pool 11 is used to micro-magnify the sample solution, and the microscope 13 can be a trinocular microscope; the camera 14 is mounted on the eyepiece of the microscope 13, and in this embodiment is specifically installed on the third eye interface of the trinocular microscope. The camera 14 is used to collect the microscopic image in the eyepiece of the microscope 13, and the microscopic image can be a video or a picture. The size of one frame / frame of the microscopic image is w*h, where w is the number of pixels in the row direction, and h is the number of pixels in the column direction. The camera 14 transmits the image data of the microscopic image through a data line, that is, each image in the microscopic image The pixel data is transmitted to the image processing unit 15, and the camera 14 can be a CMOS camera; the image processing unit 15 includes an FPGA controller 151, a first memory 152 and a second memory 153. The FPGA controller 151 includes a memory for storing computer programs and cache data and a processor for executing the computer program. When the computer program is executed by the processor, a differential dynamic microscopy image processing method based on FPGA is implemented, which configures the storage space of the memory of the FPGA controller 151, the first memory 152 and the second memory 153, and performs transmission control and calculation processing on the image data, and transmits the processed result image information to the display 16; the first memory 152 and the second memory 153 can be selected from memories with data reading and writing functions such as SDRAM chips, SRAM chips, DRAM chips, etc. In this embodiment, the first memory 152 is selected as an SDRAM chip, and the second memory 153 is selected as an SRAM chip; the display 16 displays the result image.

[0055] See also Figure 2 , which is a flow chart of the differential dynamic microscopy image processing method based on FPGA of this embodiment. The method comprises the steps of:

[0056] S1: configuring the storage space of the memory of the FPGA controller 151, the first memory 152 and the second memory 153;

[0057] S2: Acquire microscopic image information, and convert the microscopic image information into a grayscale image data stream;

[0058] S3: performing inter-frame difference processing on the grayscale image data stream to obtain a difference image data stream;

[0059] S4: performing a two-dimensional Fourier transform on the differential image data stream to obtain a corresponding Fourier spectrum image data stream, and determining the Fourier spectrum image data stream as result image information;

[0060] S5: Output the result image information.

[0061] Specifically, in step S1, a first write cache area, a first write cache area, a second write cache area, a second read cache area, a third write cache area, a third read cache area, an address cache area and a frame ID cache area are configured in the memory of the FPGA controller 151; a first address space and a second address space are configured in the first memory 152; and a plurality of image cache spaces are configured in the second memory 153. In the present embodiment, the image cache space is configured as 2.

[0062] In step S2, the microscopic image information includes image data of the microscopic image, that is, data of each pixel in the microscopic image. Figure 3 , which is a flow chart of converting the microscopic image information into a grayscale image data stream in step S2. In step S2, converting the microscopic image information into a grayscale image data stream comprises the steps of:

[0063] S21: splicing the image data of every two consecutive clock cycles in the microscopic image information to obtain a data stream in RGB565 format;

[0064] In this embodiment, the image data of each clock cycle is 8 bits, and the spliced ​​RGB565 data is 16 bits. When splicing, the data of the previous clock cycle is placed in the upper 8 bits of the 16-bit RGB565 data, and the data of the next clock cycle is placed in the lower 8 bits of the 16-bit RGB565 data.

[0065] In a preferred embodiment, before the image data in the microscopic image information is spliced, a register with a preset effective time is set. During the effective period of the register, the rising edge of the field signal output by the camera 14 is counted. The duration of a rising edge of the field signal is the transmission duration of a frame of microscopic image. At the same time, the camera 14 will also input a row signal. The duration of a rising edge of the row signal is the transmission duration of a row of microscopic image data. When the rising edge count of the field signal exceeds the preset number of times, the image data in the microscopic image information is started to be spliced. In this embodiment, the preset effective time is set to 300ms, and the preset number of times is set to 10 times.

[0066] S22: Convert the RGB565 format data stream into a YCbCr format data stream, and determine that the data of the Y channel in the YCbCr format data stream is a grayscale image data stream.

[0067] The YCbCr format data includes 8-bit Y channel data, 8-bit Cb channel data and 8-bit Cr channel data. The formula for converting the RGB565 format data into the YCbCr format data is:

[0068] Y=0.183R+0.614G+0.062B+16;

[0069] Cb=-0.101R-0.338G-0.439B+128;

[0070] Cr=0.439R-0.339G-0.040B+128.

[0071] In the formula, Y is the Y channel data value of YCbCr format data, Cb is the Cb channel data value of YCbCr format data, Cr is the Cr channel data value of YCbCr format data, R is the R channel data value of RGB format data, G is the G channel data value of RGB format data, and B is the B channel data value of RGB format data.

[0072] See also Figure 4 , which is a flow chart of performing inter-frame difference processing on the grayscale image data stream in step S3. In step S3, performing inter-frame difference processing on the grayscale image data stream includes the steps of:

[0073] S31: storing the first frame of the grayscale image data in the grayscale image data stream in the first write buffer, and writing the first frame of the grayscale image data in the first write buffer into the first address space of the first memory 152;

[0074] In this embodiment, writing the first frame of grayscale image data in the first write buffer area into the first address space of the first memory 152 specifically includes: when the amount of the grayscale image data in the first write buffer area reaches a preset burst read / write length, the grayscale image data of the burst read / write length in the first write buffer area is burst written into the first address space of the first memory 152 until all the grayscale image data of the first frame are written into the first address space. The burst read / write length can be set according to the length of a frame of image data. In this embodiment, for a frame of microscopic image with a size of 640*480, the burst read / write length is set to 256 or 128. The first address space is 0 to w*h, where w*h is the product of the length and width of the grayscale image data.

[0075] S32: when receiving the first pixel data of a new frame of the grayscale image data, reading out the first frame of the grayscale image data in the first address space and storing it in a first read buffer area;

[0076] When a new frame of grayscale image data is input, the first frame of grayscale image data is taken out from the first memory 152 to prepare for inter-frame difference calculation.

[0077] S33: reading the first frame of grayscale image data from the first read buffer area, and performing inter-frame difference calculation with the new frame of grayscale image data to obtain differential image data at a new frame moment.

[0078] Through steps S31-S33, the grayscale image data of the first frame is subtracted from the pixels at the same position of the grayscale image data of each frame after the first frame, and the coordinates of the image pixel point in the image are represented as (x, y). The pixel difference of the same pixel point at time Δt and time 0 is calculated using the inter-frame difference formula, and the inter-frame difference formula is:

[0079] D(x,y;Δt)=I(x,y;Δt)-I(x,y;0).

[0080] In step S4, the two-dimensional Fourier transform of the differential image data stream is performed by using two one-dimensional FFT IP cores of the FPGA controller 151 to respectively implement the one-dimensional Fourier transform of the column direction and the row direction of the differential image data. In this embodiment, the IP core provided by Altera is selected. The intermediate result matrix data generated between two one-dimensional Fourier transforms is cached in the first memory 152, and the single-port first memory 152 is used to read and write the intermediate result matrix data in time division multiplexing and block reading and writing. Please refer to Figure 5 , which is a schematic diagram of the process of performing two-dimensional Fourier transform on the differential image data stream in step S4, specifically comprising the following steps:

[0081] S41: performing a one-dimensional Fourier transform on the differential image data in a column direction or a row direction to obtain intermediate result matrix data;

[0082] In this embodiment, when the differential image data is subjected to a one-dimensional Fourier transform in the column direction or the row direction, the differential image data to be transformed is cached in a transform buffer area. In a specific implementation, the depth of the transform buffer area is set to 1024. When the first differential image data of each row / column is input, the row / column start signal for driving the FFT IP core to perform the transform operation is pulled high. After inputting data of a row / column transform length, the row / column end signal for stopping the FFT IP core from performing the transform operation is pulled high. The data exceeding the row / column transform length in each row / column will be truncated. The data output by the FFT IP core is aligned with the signal driving the FFT IP core to stop. The row / column transform can be set according to the size of the input image data. In this embodiment, the row / column transform length is set to 512, and the size of the intermediate result matrix of one frame output after the first one-dimensional Fourier transform is w*512 or 512*h.

[0083] S42: Generate a frame ID according to the frame time of the intermediate result matrix data and store it in the frame ID buffer area, generate an intermediate buffer address according to the frame time of the intermediate result matrix data, store the intermediate result matrix data and the corresponding intermediate buffer address in the second write buffer area, and write the intermediate result matrix data in the second write buffer area into the corresponding image buffer space in the second memory 153 according to the intermediate buffer address in the second write buffer area;

[0084] When the differential image data is subjected to a one-dimensional Fourier transform in the column direction, the intermediate result matrix data obtained is written into the image cache space in sequence by column; when the differential image data is subjected to a one-dimensional Fourier transform in the row direction, the intermediate result matrix data obtained is written into the image cache space in sequence by row.

[0085] In this embodiment, while the intermediate result matrix data and the corresponding intermediate cache address are stored in the second write cache area, the stored intermediate result matrix or intermediate cache address is counted for storage data. When the storage data count reaches a preset storage data value, the intermediate result matrix data in the second write cache area is started to be written into the corresponding image cache space. While the intermediate result matrix data in the second write cache area is written into the corresponding image cache space, the write data is counted starting from 0. When the write data count reaches a preset write data value, the write data into the image cache space is stopped. Please refer to Figure 6, which is an exemplary state machine flow chart of the read and write control of the second memory 153 in a specific implementation. In the idle state and write SRAM state conversion process, the storage data count of the second write buffer is defined as wcmd_fifo_rcnt, the write data count of the second write buffer is defined as wcnt, the preset storage data value is set to 256, and the preset write data value is set to 255. Then, when the processor of the FPGA controller 151 is in the idle state, when wcmd_fifo_rcnt≥256, it enters the write SRAM state, at which time the amount of data of the intermediate result matrix stored in the second write buffer is 256; when the processor of the FPGA controller 151 is in the write SRAM state, when wcnt==255, it enters the idle state, at which time 256 intermediate result matrix data are read from the second write buffer and written into the image cache space.

[0086] S43: Generate a corresponding intermediate buffer address according to the frame ID in the frame ID buffer and store it in the address buffer, read out the intermediate result matrix data in the corresponding image buffer space according to the intermediate buffer address in the address buffer and store it in the second read buffer;

[0087] If the first one-dimensional Fourier transform direction of the differential image data is the column direction, the intermediate result matrix data is read out from the image cache space by row; if the first one-dimensional Fourier transform direction of the differential image data is the row direction, the intermediate result matrix data is read out from the image cache space by row.

[0088] In this embodiment, when the frame ID data starts to be stored in the frame ID buffer, the corresponding intermediate buffer address is generated according to the frame ID in the frame ID buffer. While the corresponding intermediate buffer address is generated according to the frame ID in the frame ID buffer, according to the row data and column data counts of the differential image data, when the row data count and the column data count reach the preset values ​​respectively, the generation of the intermediate buffer address of one frame of intermediate result matrix data is completed, and the generation of the intermediate buffer address is stopped at this time. Please refer to Figure 7, which is an exemplary state machine flow chart of intermediate cache address generation control in a specific implementation. In the switching between the idle state and the intermediate cache address generation state, the row data count is defined as addr_hcnt, the column data count is defined as addr_vcnt, the preset row data value is 511, the preset column data value is 479, and the storage state of the address cache area is defined as raddr_fifo_full, with values ​​of 0 and 1 indicating that the storage space is not full and full. The processor of the FPGA controller 151 is in the idle state and enters the SRAM address generation state when the address cache area is not empty; in the intermediate cache address generation state, when (raddr_fifo_afull==0)&(addr_hcnt==511)&(addr_vcnt==479), it enters the intermediate cache address generation end state and returns to the idle state.

[0089] In this embodiment, while the corresponding intermediate cache address is generated and stored in the address cache area, a storage data count is performed on the intermediate cache address stored in the address cache area. When the storage data count exceeds a preset storage data value and the storage state of the second read cache area is that the storage space is not full, the intermediate result matrix data in the corresponding image cache space is read out. While reading out the intermediate result matrix data in the corresponding image cache space, a storage data count is performed on the intermediate result matrix or intermediate cache address stored in the second write cache area. If the storage data count reaches a preset storage data value, the reading of the intermediate result matrix data from the image cache space will be stopped; at the same time, the read data is counted starting from 0. When the read data count reaches a preset read data value, the reading of the intermediate result matrix data from the image cache space will be stopped. Please refer again. Figure 6In the idle state and read SRAM state conversion process, the storage data count of the address buffer is defined as raddr_fifo_rcnt, the preset storage data value is set to 480, the storage state of the second read buffer is defined as rdata_fifo_full, with values ​​of 0 and 1 indicating that the storage space is not full and full, the read data count is defined as rcnt, and the preset read data value is set to 479. Then, the processor of the FPGA controller 151 is in the idle state, and when (raddr_fifo_rcnt≥480)&&(rdata_fifo_full==0), it enters the read SRAM state. At this time, the amount of data of the intermediate cache address stored in the address buffer is 256; the processor of the FPGA controller 151 is in the read SRAM state, and when rcnt==479, it enters the idle state. At this time, 480 intermediate result matrix data are read from the image cache space and written into the second read buffer. In the state conversion process of writing SRAM and reading SRAM, the processor of the FPGA controller 151 is in the state of reading SRAM, and enters the state of writing SRAM when wcmd_fifo_rcnt≥256.

[0090] S44: performing a one-dimensional Fourier transform in another direction on the intermediate result matrix data in the second read buffer area to obtain two-dimensional transformation result data;

[0091] In this embodiment, the row / column transformation length is set to 512, so the size of a frame of two-dimensional transformation result data is 512*512.

[0092] S45: Calculate the square of the amplitude of the absolute value of the two-dimensional transformation result data, and determine it as Fourier spectrum image data.

[0093] See also Figure 8 , which is a schematic diagram of the process of outputting the result image information in step S5, and step S5 specifically includes the steps of:

[0094] S51: storing the result image information in the third write buffer area, and writing the result image information in the third write buffer area into the second address space;

[0095] S52: Read out the result image information in the second address space and store it in a third read buffer area, and output the result image information in the third read buffer area.

[0096] Based on the same inventive concept, the present invention also provides a differential dynamic microscopic image processing device based on FPGA, please refer to Fig. 9, which is a structural schematic diagram of the device, the device includes a storage space configuration unit 21, an image conversion unit 22, an inter-frame difference unit 23, a two-dimensional Fourier transform unit 24 and an output unit 25, wherein the storage space configuration unit 21 is used to configure the memory of the FPGA controller 151, the storage space of the first memory 152 and the second memory 153; the image conversion unit 22 is used to obtain microscopic image information and convert the microscopic image information into a grayscale image data stream; the inter-frame difference unit 23 is used to perform inter-frame difference processing on the grayscale image data stream to obtain a differential image data stream; the two-dimensional Fourier transform unit 24 is used to perform a two-dimensional Fourier transform on the differential image data stream to obtain a corresponding Fourier spectrum image data stream, and determine that the Fourier spectrum image data stream is the result image information; the output unit 25 is used to output the result image information.

[0097] Specifically, the image conversion unit 22 includes a data splicing module 221 and a data conversion module 222. The data splicing module 221 is used to splice the image data of every two consecutive clock cycles in the microscopic image information to obtain an RGB565 format data stream; the data conversion module 222 is used to convert the RGB565 format data stream into a YCbCr format data stream, and determine that the data of the Y channel in the YCbCr format data stream is a grayscale image data stream.

[0098] The inter-frame differential unit 23 includes a first write control module 231, a first read control module 232 and a differential calculation module 233. The first write control module 231 is used to store the first frame of the grayscale image data in the grayscale image data stream in the first write buffer area, and write the first frame of the grayscale image data in the first write buffer area into the first address space of the first memory 152; the first read control module 232 is used to read out the first frame of the grayscale image data in the first address space and store it in the first read buffer area when receiving the first pixel data of a new frame of the grayscale image data; the differential calculation module 233 is used to read the first frame of the grayscale image data from the first read buffer area, and perform inter-frame differential calculation with the new frame of grayscale image data to obtain differential image data at a new frame moment.

[0099] The two-dimensional Fourier transform unit 24 includes a first transform module 241, a second write control module 242, a second read control module 243, a second transform module 244 and a spectrum image calculation module 245. The first transform module 241 is used to perform a one-dimensional Fourier transform in the column direction or the row direction on the differential image data to obtain intermediate result matrix data; the second write control module 242 is used to generate a frame ID according to the frame time of the intermediate result matrix data and store it in the frame ID buffer area, generate an intermediate buffer address according to the frame time of the intermediate result matrix data, store the intermediate result matrix data and the corresponding intermediate buffer address in the second write buffer area, and generate a frame ID according to the frame time of the intermediate result matrix data. The intermediate result matrix data in the second write buffer area is written into the corresponding image buffer space according to the storage address; the second read control module 243 is used to generate the corresponding intermediate buffer address according to the frame ID in the frame ID buffer area and store it in the address buffer area, and read out the intermediate result matrix data in the corresponding image buffer space according to the intermediate buffer address in the address buffer area and store it in the second read buffer area; the second transformation module 244 is used to perform one-dimensional Fourier transform in another direction on the intermediate result matrix data in the second read buffer area to obtain two-dimensional transformation result data; the spectrum image calculation module 245 is used to calculate the square of the amplitude of the absolute value of the two-dimensional transformation result data, and determine it as Fourier spectrum image data.

[0100] The output unit 25 includes a third write control module 251 and a third read control module 252. The third write control module 251 is used to store the result image information in the third write cache area, and write the result image information in the third write cache area into the second address space; the third read control module 252 is used to read out the result image information in the second address space according to the output timing signal and store it in the third read cache area, and output the result image information in the third read cache area.

[0101] As for the device embodiment, since it basically corresponds to the above-mentioned image processing method embodiment, the specific implementation details of the device embodiment are the same as the implementation details disclosed in the embodiment of the image processing method of the present invention, and are not repeated here. The device embodiment described above is only illustrative, wherein the units / modules described as separate components may or may not be physically separated, and the components shown as units / modules may or may not be physical units / modules.

[0102] A comparative experiment was conducted on the differential dynamic microscopic image processing method based on FPGA of the present invention and a MATLAB differential dynamic microscopic image processing method based on ordinary computer. The two methods were used to process the microscopic image respectively, and the clock frequency was set to 50Mhz. The experimental results showed that for the inter-frame difference calculation of a frame of microscopic image, the processing time of the method of the present invention was about 12ms, and the processing time of the method based on ordinary computer was about 300ms. It can be seen that the processing speed of the method of the present invention was increased by about 25 times; for the two-dimensional Fourier transform calculation of a frame of microscopic image, the processing time of the method of the present invention was about 4.9ms, and the processing time of the method based on ordinary computer was about 15ms. It can be seen that the processing speed of the method of the present invention was increased by about 3 times. Compared with the prior art, the differential dynamic microscopic image processing method based on FPGA of the present invention reasonably and efficiently utilizes the storage resources inside and outside the FPGA chip, which can meet the processing requirements of large data volume and high real-time requirements, accelerate the calculation speed of the core algorithm module, reduce the system volume, and have strong parallel capability.

[0103] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and the present invention is also intended to include these modifications and modifications.

Claims

1. A differential dynamic microscopy image processing method based on FPGA, characterized in that: Includes steps: A first write cache area, a first read cache area, a second write cache area, a second read cache area, an address cache area, and a frame ID cache area are configured in the memory within the FPGA chip, a first address space is configured in the first memory outside the FPGA chip, and a plurality of image cache spaces are configured in the second memory outside the FPGA chip; Acquire microscopic image information, and convert the microscopic image information into a grayscale image data stream; storing the first frame of the grayscale image data in the grayscale image data stream in the first write buffer area, and writing the first frame of the grayscale image data in the first write buffer area into the first address space; When receiving the first pixel data of a new frame of the grayscale image data, reading out the first frame of grayscale image data in the first address space and storing it in the first read buffer area; Reading the first frame of grayscale image data from the first read buffer area, and performing inter-frame difference calculation with the new frame of grayscale image data to obtain differential image data at a new frame moment; Performing a two-dimensional Fourier transform on the differential image data to obtain corresponding Fourier spectrum image data, and determining the Fourier spectrum image data as result image information; Performing a two-dimensional Fourier transform on the differential image data to obtain corresponding Fourier spectrum image data comprises the steps of: Performing a one-dimensional Fourier transform on the differential image data in a column direction or a row direction to obtain intermediate result matrix data; Generate a frame ID according to the frame time of the intermediate result matrix data and store it in the frame ID buffer area, generate an intermediate buffer address according to the frame time of the intermediate result matrix data, store the intermediate result matrix data and the corresponding intermediate buffer address in the second write buffer area, and write the intermediate result matrix data in the second write buffer area into the corresponding image buffer space according to the intermediate buffer address in the second write buffer area; Generate a corresponding intermediate buffer address according to the frame ID in the frame ID buffer, store the intermediate buffer address in the address buffer, read out the corresponding intermediate result matrix data in the image buffer space according to the intermediate buffer address in the address buffer, and store the intermediate result matrix data in the second read buffer; Performing a one-dimensional Fourier transform in another direction on the intermediate result matrix data in the second read buffer area to obtain two-dimensional transformation result data; Calculating the square of the amplitude of the absolute value of the two-dimensional transformation result data and determining it as Fourier spectrum image data; The storage space configuration unit is used to configure a second write cache area, a second read cache area, an address cache area and a frame ID cache area in the memory within the FPGA chip, and configure multiple image cache spaces in the second memory outside the FPGA chip; The two-dimensional Fourier transform unit includes a first transform module, a second write control module, a second read control module, a second transform module and a spectrum image calculation module, wherein the first transform module is used to perform a one-dimensional Fourier transform in a column direction or a row direction on the differential image data to obtain intermediate result matrix data; the second write control module is used to generate a frame ID according to a frame time of the intermediate result matrix data and store it in the frame ID buffer area, generate an intermediate buffer address according to the frame time of the intermediate result matrix data, store the intermediate result matrix data and the corresponding intermediate buffer address in the second write buffer area, and write the intermediate result matrix data in the second write buffer area into the corresponding image buffer space according to the intermediate buffer address in the second write buffer area; The second read control module is used to generate a corresponding intermediate buffer address according to the frame ID in the frame ID buffer area, store the intermediate buffer address in the address buffer area, read out the corresponding intermediate result matrix data in the image buffer space according to the intermediate buffer address in the address buffer area, and store it in the second read buffer area; The second transformation module is used to perform a one-dimensional Fourier transform in another direction on the intermediate result matrix data in the second read buffer area to obtain two-dimensional transformation result data; The spectrum image calculation module is used to calculate the square of the amplitude of the absolute value of the two-dimensional transformation result data and determine it as Fourier spectrum image data.

2. The method according to claim 1, characterized in that Writing the first frame of grayscale image data in the first write buffer area into the first address space includes the following steps: when the amount of the grayscale image data in the first write buffer area reaches a preset burst read / write length, burst writing the grayscale image data of the burst read / write length in the first write buffer area into the first address space until all the grayscale image data of the first frame are written into the first address space.

3. The method according to claim 2, characterized in that While storing the intermediate result matrix data and the corresponding intermediate cache address in the second write cache area, the method further includes the steps of: counting the intermediate result matrix or the intermediate cache address stored in the second write cache area, and when the count reaches a preset value, starting to write the intermediate result matrix data in the second write cache area into the corresponding image cache space according to the intermediate cache address in the second write cache area; While writing the intermediate result matrix data in the second write cache area into the corresponding image cache space, it also includes the step of: counting the intermediate result matrix data written into the image cache space, and when the count reaches a preset value, stopping writing the intermediate result matrix data in the second write cache area into the corresponding image cache space.

4. The method according to claim 1, characterized in that: While storing the intermediate cache address in the address cache area, the method further includes the steps of: counting the intermediate cache address stored in the address cache area, and when the count exceeds a preset value and the storage space of the second read cache area is not full, starting to read out the intermediate result matrix data in the corresponding image cache space according to the intermediate cache address in the address cache area; While reading out the intermediate result matrix data in the corresponding image cache space and storing it in the second read cache area, it also includes the steps of: counting the intermediate result matrix data read out from the corresponding image cache space, and when the count reaches a preset value, stopping reading out the intermediate result matrix data in the corresponding image cache space; counting the intermediate result matrix or the intermediate cache address stored in the second write cache area, and if the count reaches a preset value, stopping reading out the intermediate result matrix data in the corresponding image cache space.

5. A differential dynamic microscopy image processing device based on FPGA for implementing the method according to any one of claims 1 to 4, characterized in that: include: A storage space configuration unit, configured to configure a first write cache area and a first read cache area in the memory within the FPGA chip, and to configure a first address space in a first memory outside the FPGA chip; An image conversion unit, used for acquiring microscopic image information and converting the microscopic image information into a grayscale image data stream; an inter-frame difference unit, comprising a first write control module, a first read control module and a difference calculation module, wherein the first write control module is used to store the first frame of the grayscale image data in the grayscale image data stream in the first write buffer area, and write the first frame of the grayscale image data in the first write buffer area into the first address space; The first read control module is used for reading out the first frame of grayscale image data in the first address space and storing it in the first read buffer area when receiving the first pixel data of a new frame of grayscale image data; The difference calculation module is used to read the first frame of grayscale image data from the first read buffer area, and perform inter-frame difference calculation with the new frame of grayscale image data to obtain the difference image data of the new frame; The two-dimensional Fourier transform unit is used to perform a two-dimensional Fourier transform on the differential image data to obtain corresponding Fourier spectrum image data, and determine the Fourier spectrum image data as result image information.

6. The device according to claim 5, characterized in that The first write control module writes the first frame of grayscale image data of the first write buffer area into the first address space, including the steps of: when the data amount of the grayscale image data in the first write buffer area reaches a preset burst read / write length, burst writing the grayscale image data of the burst read / write length in the first write buffer area into the first address space until all the grayscale image data of the first frame are written into the first address space.

7. The device according to claim 6, characterized in that: The second write control module stores the intermediate result matrix data and the corresponding intermediate cache address in the second write cache area, and counts the intermediate result matrix or the intermediate cache address stored in the second write cache area. When the count reaches a preset value, the intermediate result matrix data in the second write cache area starts to be written into the corresponding image cache space according to the intermediate cache address in the second write cache area; The second write control module counts the intermediate result matrix data written into the image cache space while writing the intermediate result matrix data in the second write cache area into the corresponding image cache space, and stops writing the intermediate result matrix data in the second write cache area into the corresponding image cache space when the count reaches a preset value; The second read control module stores the intermediate cache address in the address cache area and counts the intermediate cache address stored in the address cache area. When the count exceeds a preset value and the storage space of the second read cache area is not full, the intermediate result matrix data in the image cache space corresponding to the intermediate cache address in the address cache area is read out. The second read control module reads out the intermediate result matrix data in the corresponding image cache space and stores it in the second read cache area, while counting the intermediate result matrix data read out in the corresponding image cache space. When the count reaches a preset value, the reading of the intermediate result matrix data in the corresponding image cache space is stopped; the intermediate result matrix or the intermediate cache address stored in the second write cache area is counted. If the count reaches a preset value, the reading of the intermediate result matrix data in the corresponding image cache space is stopped.

8. A differential dynamic microscopic image processing system based on FPGA, comprising a microscope, a camera and an image processing unit, wherein the camera collects microscopic images in the microscope and transmits them to the image processing unit, characterized in that: The image processing unit includes an FPGA controller, a first memory and a second memory. The FPGA controller includes a memory for storing computer programs and cache data and a processor for executing the computer program. When the computer program is executed by the processor, the steps of the method described in any one of claims 1 to 4 are implemented.

Images