Differential dynamic microscopic image processing method and system based on ZYNQ

Abstract

The application discloses a differential dynamic microscopic image processing method and system based on ZYNQ. The system and method realize the logic circuit implementation of the differential dynamic microscopic image processing method by means of the high-speed parallelism and complex operation processing capacity of ZYNQ, greatly improve the experiment speed and reduce the volume of the computing device. Compared with other implementation methods, the application uses a CMOS camera and a ZYNQ chip to complete the operation of the differential dynamic microscopic image processing method algorithm of the input image, simplifies the system structure, accelerates the operation process, and realizes the miniaturization and instrumentation of the algorithm device.

Description

Technical Field

[0001] This invention relates to the field of particle microscopic image processing technology, and in particular to a differential dynamic microscopic image processing method and system based on ZYNQ. Background Technology

[0002] The development of nanomaterials technology is closely linked to the development of nanoparticle measurement and characterization techniques. Differential dynamic microscopy plays a crucial role in the study of nanoparticle dynamics.

[0003] Based on research findings both domestically and internationally, experimental methods for studying the Brownian motion of particles mainly include two types: particle tracking and dynamic light scattering. Particle tracking requires optically resolvable particles, thus imposing significant limitations regarding solution concentration and particle size. Dynamic light scattering, on the other hand, is highly susceptible to the effects of multiple scattering caused by high concentrations. While diffuse wave spectroscopy and fiber quasi-elastic scattering techniques can directly analyze multiple scattered light to overcome these limitations, they also introduce drawbacks such as complex experimental setups.

[0004] Researchers typically use ordinary computers to process particle microscopic images, which is highly versatile. However, because CPUs (Central Processing Units) process serially, they suffer from long computation times when processing a large number of microscopic images (Chen Yicun, Zhang Wenliang, Qiu Jian, Luo Kaiqing, Peng Li, Han Peng. Experimental study on Brownian motion based on differential dynamic microscopy [J]. China Powder Technology, 2017, 23(02):30-34. DOI:10.13732 / j.issn.1008-5548.207.02.006). ASIC-based processing methods, on the other hand, suffer from poor flexibility and high cost. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a differential dynamic microscopy image processing method and system based on ZYNQ. This system uses ZYNQ for computational processing and has advantages such as high flexibility, small size, low cost, and low power consumption, making it highly suitable for the implementation of differential dynamic microscopy image processing technology.

[0006] The objective of this invention is achieved by at least one of the following technical solutions.

[0007] The ZYNQ-based differential dynamic microscopic image processing method includes the following steps:

[0008] S1. Acquiring initial microscopic images: The ZYNQ (Zynq-7000 All Programmable SoC) is configured with a CMOS industrial camera via the SCCB protocol. The initialized CMOS industrial camera acquires microscopic image information of the sample solution through a microscope. The transmission bit width of the CMOS industrial camera is 8 bits.

[0009] S2. Grayscale conversion: Because the 16-bit RGB565 output by the camera is sent in two parts, before proceeding to the next step, it is necessary to concatenate the 16-bit RGB565 into 24-bit RGB data, and then remap the 24-bit RGB format data into YCbCr format data to obtain the grayscale pixel information of the Y channel.

[0010] S3, Inter-frame difference: After the camera is initialized, the first frame image data is taken and stored in RAM. When each frame image from the second frame onwards is passed in, the first frame image data is used to subtract the corresponding pixel data of each frame image from the second frame onwards to obtain the image difference data matrix.

[0011] S4. Image Fourier Transform: Before performing the Fourier transform, the frame image size is cropped to a power of 2 to facilitate subsequent Fourier transform operations. Based on the relative independence of row and column transformations in the image Fourier transform, the image Fourier transform is split into row Fourier transform and column Fourier transform. After performing a row Fourier transform on the image data, the transformed row data is stored in DDR. When reading later, the data is read column by column, and the column data is transformed again. Finally, the Fourier transform result data matrix of the image is output.

[0012] S5. Radial averaging: After obtaining the Fourier transform result data matrix, the values ​​of points along a set radius are sampled. Because of isotropy, the sampled values ​​are consistent. The radial averaging result is obtained by averaging the uniformly distributed points.

[0013] Furthermore, in step S1, the microscopic image information of the sample solution is transmitted from the microscope to the CMOS industrial camera, and then transmitted to the ZYNQ via an eight-bit data transmission line in the DVP video stream format. The camera's shooting settings are configured as follows: 640*480 pixels, 30 frames per second, RGB565 format.

[0014] Furthermore, in step S2, since the hardware connection data bit width of the camera is only eight bits, and the RGB565 format video stream transmission method is to split the RGB data of one pixel into two pixel clocks for transmission, the RGB565 is first split and concatenated into an 8-bit data format to facilitate subsequent grayscale conversion.

[0015] Since subsequent processing only requires the grayscale pixel information of the Y channel of the image, in order to simplify the operation of ZYNQ and reduce resource waste, the input RGB format video stream data is first converted to YCbCr format, and only the grayscale value Y in the YCbCr format is taken.

[0016] Furthermore, in step S3, after the camera completes initialization, ZYNQ begins collecting the first frame data from the video stream and stores it in RAM. When the first frame data is stored and the second frame data begins to be transmitted, each pixel starting from the first pixel of the second frame is subtracted from the corresponding pixel of the first frame stored in RAM. During the subtraction, the two pixels to be subtracted need to be compared to ensure that the larger number is always subtracted from the smaller number. This results in obtaining a frame difference image video stream transmitted in DVP format.

[0017] Further, in step S4, the DVP format frame difference image video stream obtained in step S3 is cropped in pixel size from 640*480 pixels to 256*256 pixels; then the DVP video stream format is converted to AXI-stream format so that Fourier transform can be performed.

[0018] Furthermore, based on the row-column independence in the image Fourier transform, a single image Fourier transform is split into a row transform and a column transform. The data after converting the video stream format directly enters the row Fourier transform module to perform Fourier transform on the row data, and Fourier transform is performed on 256 rows of data in sequence. The calculated data is stored in DDR. Unlike the general sequential storage method, in order to facilitate subsequent column-by-column data reading, the data of each row is stored in a column in sequence. The writing method is that the address jumps according to the row data length. After completing a row, the address is reset and incremented by one, thus completing the transpose process of the image data.

[0019] Furthermore, after the data processed by line transformation is stored in a full frame, it will be stored in a new storage space to avoid mutual interference between read and write data operations, so as to avoid image tearing.

[0020] As the second frame of data begins to be stored, the column Fourier transform module also begins to read the stored data. Since the previous data has already undergone image offset processing, the column data can be read directly in sequence. The length of one row is used as the reading length of one burst read, and the reading is divided into 256 reads to complete the reading of the column data and the Fourier transform processing.

[0021] Furthermore, to avoid image tearing, an offset address is added to both the write and read addresses. The size of the offset address is the frame number multiplied by the size of a single frame's storage space. Since the read and write speeds remain consistent, allocating three frame numbers is sufficient to prevent image tearing.

[0022] Further, in step S5, firstly, the single-frame image calculated in step S4 has rotation invariance, that is, with the center of the image as the point, the values ​​within the same radius are the same; eight points that are uniformly distributed and mirror-symmetric within the same radius are selected as the radially averaged sample values; because of the digital image, the selected points may not fall exactly at the center of the pixel. In order to ensure the accuracy of the implementation, a bilinear interpolation is performed on the selected points, that is, for a selected point, its distance to the four nearest points is used as the weight and multiplied by the values ​​corresponding to the four nearest points to obtain the value of the selected point.

[0023] The implementation of a typical bilinear interpolation algorithm requires real-time determination of the distance between a point and its four nearest neighbors, which necessitates complex data scheduling. However, since the selected points are eight points with defined coordinates, in order to simplify the operation process and save system resources, the distances (i.e., weights) between the selected points and their four nearest neighbors can be stored in ROM in advance. When radial averaging is required, these weights are read out to obtain the difference between the selected points. After obtaining all the points, the interpolation is performed, and the values ​​are added together and multiplied to obtain the radial averaging result.

[0024] The differential dynamic microscopic image processing system based on ZYNQ includes a light source, sample solution, microscope, CMOS camera, and ZYNQ image processing module;

[0025] The light source is mounted on the side of the sample solution, and the emitted light shines on the sample solution. The light is magnified by a microscope, and then a CMOS camera mounted on the microscope acquires the microscopic image. A sensor array detects the light signal magnified by the microscope and converts it into an electrical signal to input the microscopic image information into the ZYNQ image processing module that is spliced ​​together with the CMOS camera.

[0026] The ZYNQ image processing module includes a camera access module based on the SCCB protocol, an AXI-stream video stream format conversion module, a cropping module, a grayscale conversion module, an inter-frame difference module, an image Fourier transform module, and a radial averaging module.

[0027] The image Fourier transform module includes a transpose scheduling module, a DDR module, and a Fourier transform calculation module; the camera access module based on the SCCB protocol completes the configuration of the camera and video stream access; the AXI-stream video stream format conversion module completes the conversion from DVP video format to AXI-stream format; the cropping module completes the cropping of the image size; the grayscale conversion module obtains the grayscale value in the pixel; the inter-frame difference module completes the difference processing between frame data; the image Fourier transform module completes the Fourier transform of the image; and the radial averaging module completes the interpolation and weighted summation of the sampling points to finally obtain the radial average result.

[0028] Compared with the prior art, the advantages of the present invention are:

[0029] Compared to previous technologies, this invention utilizes a CMOS camera and a ZYNQ chip to perform the calculations for the differential dynamic microscopy image processing algorithm on the input image. This approach simplifies the system structure and accelerates the computation process, enabling the miniaturization and instrumentation of the algorithm device. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the differential dynamic microscopic image processing system based on ZYNQ in an embodiment of the present invention;

[0031] Figure 2 This is a flowchart illustrating the differential dynamic microscopic image processing system based on ZYNQ in an embodiment of the present invention.

[0032] Figure 3 This is a structural diagram of the frame difference module in an embodiment of the present invention;

[0033] Figure 4 This is a structural diagram of the image Fourier transform module in an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of data storage during the transpose operation in an embodiment of the present invention;

[0035] Figure 6 This is a flowchart illustrating the process of reading data from multiple storage spaces in an embodiment of the present invention.

[0036] Figure 7 This is a schematic diagram of the weighted summation operation in an embodiment of the present invention;

[0037] Figure 8 This is a flowchart illustrating the steps of the differential dynamic microscopic image processing method based on ZYNQ in an embodiment of the present invention. Detailed Implementation

[0038] To more clearly illustrate the objectives, technical solutions, and advantages of the present invention, the specific implementation methods of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0039] Example:

[0040] ZYNQ-based differential dynamic microscopic image processing methods, such as Figure 8 As shown, it includes the following steps:

[0041] S1. Acquiring initial microscopic images: The ZYNQ (Zynq-7000 All Programmable SoC) is configured with the CMOS industrial camera via the SCCB protocol. The initialized CMOS industrial camera acquires microscopic image information of the sample solution through a microscope. The transmission bit width of the CMOS industrial camera is 8 bits.

[0042] The microscopic image information of the sample solution is transmitted from the microscope to the CMOS industrial camera, and then transmitted to the ZYNQ in the form of a DVP video stream via an eight-bit data transmission line. The camera's shooting settings are configured as follows: 640*480 pixels, 30 frames per second, RGB565 format.

[0043] S2. Grayscale conversion: Because the 16-bit RGB565 output by the camera is sent in two parts, before proceeding to the next step, it is necessary to concatenate the 16-bit RGB565 into 24-bit RGB data, and then remap the 24-bit RGB format data into YCbCr format data to obtain the grayscale pixel information of the Y channel.

[0044] Firstly, because the hardware connection data width of the camera is only eight bits, the RGB565 format video stream transmission method is to split the RGB data of one pixel into two pixel clocks for transmission. Therefore, the RGB565 is first split and concatenated into an 8-bit data format to facilitate subsequent grayscale conversion.

[0045] Since subsequent processing only requires the grayscale pixel information of the Y channel of the image, in order to simplify the operation of ZYNQ and reduce resource waste, the input RGB format video stream data is first converted to YCbCr format, and only the grayscale value Y in the YCbCr format is taken.

[0046] S3, Inter-frame difference: After the camera is initialized, the first frame image data is taken and stored in RAM. When each frame image from the second frame onwards is passed in, the first frame image data is used to subtract the corresponding pixel data of each frame image from the second frame onwards to obtain the image difference data matrix.

[0047] After the camera completes initialization, ZYNQ begins collecting the first frame of video stream data and storing it in RAM. Once the first frame data is stored, when the second frame data begins to be received, each pixel starting from the first pixel of the second frame is subtracted from the corresponding pixel in the first frame stored in RAM. During the subtraction, the two pixels to be subtracted are compared to ensure that the larger number is always subtracted from the smaller number. This process yields a frame difference image video stream transmitted in DVP format.

[0048] S4. Image Fourier Transform: Before performing the Fourier transform, the frame image size is cropped to a power of 2 to facilitate subsequent Fourier transform operations. Based on the relative independence of row and column transformations in the image Fourier transform, the image Fourier transform is split into row Fourier transform and column Fourier transform. After performing a row Fourier transform on the image data, the transformed row data is stored in DDR. When reading later, the data is read column by column, and the column data is transformed again. Finally, the Fourier transform result data matrix of the image is output.

[0049] The DVP format frame difference image video stream obtained in step S3 is cropped in pixel size from 640*480 pixels to 256*256 pixels; then the DVP video stream format is converted to AXI-stream format so that Fourier transform can be performed.

[0050] Furthermore, based on the row-column independence in the image Fourier transform, a single image Fourier transform is split into a row transform and a column transform. The data after converting the video stream format directly enters the row Fourier transform module to perform Fourier transform on the row data, and Fourier transform is performed on 256 rows of data in sequence. The calculated data is stored in DDR. Unlike the general sequential storage method, in order to facilitate subsequent column-by-column data reading, the data of each row is stored in a column in sequence. The writing method is that the address jumps according to the row data length. After completing a row, the address is reset and incremented by one, thus completing the transpose process of the image data.

[0051] Furthermore, after the data processed by line transformation is stored in a frame, it will be stored in a new storage space to avoid mutual interference between read and write data operations, which would cause image tearing.

[0052] As the second frame of data begins to be stored, the column Fourier transform module also begins to read the stored data. Since the previous data has already undergone image offset processing, the column data can be read directly in sequence. The length of one row is used as the reading length of one burst read, and the reading is divided into 256 reads to complete the reading of the column data and the Fourier transform processing.

[0053] Furthermore, to avoid image tearing, an offset address is added to both the write and read addresses. The size of the offset address is the frame number multiplied by the size of a single frame's storage space. Since the read and write speeds remain consistent, allocating three frame numbers is sufficient to prevent image tearing.

[0054] S5, Radial Average: After obtaining the Fourier transform result data matrix, the values ​​of points along a set radius are sampled. Because of isotropy, the sampled values ​​are consistent. The radial average result is obtained by averaging the uniformly distributed points.

[0055] First, the single-frame image calculated in step S4 is rotationally invariant, meaning that the values ​​within the same radius with the image center as the point are the same. Eight points that are uniformly distributed and mirror-symmetric within the same radius are selected as radially averaged sample values. Because of the digital image, the selected points may not fall exactly at the center of the pixel. To ensure accuracy, a bilinear interpolation is performed on the selected points. That is, for a selected point, its distance to the four nearest points is used as a weight to multiply the values ​​corresponding to the four nearest points to obtain the value of the selected point.

[0056] The implementation of a typical bilinear interpolation algorithm requires real-time determination of the distance between a point and its four nearest neighbors, which necessitates complex data scheduling. However, since the selected points are eight points with defined coordinates, in order to simplify the operation process and save system resources, the distances (i.e., weights) between the selected points and their four nearest neighbors can be stored in ROM in advance. When radial averaging is required, these weights are read out to obtain the difference between the selected points. After obtaining all the points, the interpolation is performed, and the values ​​are added together and multiplied to obtain the radial averaging result.

[0057] ZYNQ-based differential dynamic microscopic image processing system, such as Figure 1 As shown, it includes a light source R1, a sample solution R2, a microscope R3, a CMOS camera R4, and a ZYNQ image processing module R5;

[0058] In one embodiment, R1 can be white light as the light source, used to illuminate the sample solution in R2; R3 is a commonly used trinocular microscope; R4 is a conventional OV5640 monocular industrial camera, mounted on the third eye interface of the trinocular microscope, used to receive magnified images; R5 is the xc7z020clg400 chip from the ZYNQ-7 series.

[0059] The light source is mounted on the side of the sample solution, and the emitted light shines on the sample solution. The light is magnified by the microscope, and then the microscopic image is acquired by the CMOS camera mounted on the microscope. The light signal magnified by the microscope is converted into an electrical signal and the microscopic image information is input into the ZYNQ image processing module that is spliced ​​together with the CMOS camera.

[0060] like Figure 2 As shown, the ZYNQ image processing module includes a camera access module based on the SCCB protocol, an AXI-stream video stream format conversion module, a cropping module, a grayscale conversion module, an inter-frame difference module, an image Fourier transform module, and a radial averaging module.

[0061] The image Fourier transform module includes a transpose scheduling module, a DDR module, and a Fourier transform calculation module; the camera access module based on the SCCB protocol completes the configuration of the camera and video stream access; the AXI-stream video stream format conversion module completes the conversion from DVP video format to AXI-stream format; the cropping module completes the cropping of the image size; the grayscale conversion module obtains the grayscale value in the pixel; the inter-frame difference module completes the difference processing between frame data; the image Fourier transform module completes the Fourier transform of the image; and the radial averaging module completes the interpolation and weighted summation of the sampling points to finally obtain the radial average result.

[0062] In one embodiment, the OV5640 is selected as the image acquisition sensor in the camera module. The OV5640 is a 1 / 4-inch single-chip image sensor with a photosensitive array of 25,921,944 pixels (i.e., 5 million pixels), capable of acquiring images at a maximum resolution of 15fps QSXVGA (25,921,944 pixels) or 90fps VGA (640*480). The OV5640 camera uses SCCB (Serial Camera Control Bus) for camera configuration. SCCB is a three-wire serial bus defined and developed by OV (short for OmniVision). This protocol allows adjustment of most camera functions, including image data format, resolution, and image processing parameters. In engineering practice, the more commonly used I2C protocol can be used for initialization because SCCB is compatible with the I2C bus and does not affect practical use. In this project, the camera is configured to operate at 30 frames per second, with a pixel resolution of 640*480 and RGB565 output mode. The OV5640 module uses DVP to transmit video. PCLK is the pixel clock, HREF is the horizontal sync signal (video data is valid when HREF is high), and VSYNC is the vertical sync signal. The data bus has 10 bits, but only 8 bits are used.

[0063] In one embodiment, the data splicing module is configured to output RGB565 data. Because the data bus is 8-bit, one pixel's data is transmitted every two clock cycles, and then spliced ​​into complete pixel data at the ZYNQ receiver. In this case, the output RGB565 is one pixel every two clock cycles, with one clock cycle containing unused data. A valid output signal (de_o) is used to indicate whether the pixel data is valid.

[0064] In one embodiment, the grayscale conversion module performs an RGB to YCbCr color space conversion to obtain grayscale data for subsequent inter-frame difference calculations. In YCbCr, Y refers to the luminance component, Cb to the blue chrominance component, and Cr to the red chrominance component. The human eye is more sensitive to the Y component of video. The specific operation of converting from RGB to YCbCr is based on the grayscale conversion formula:

[0065] Y = 0.299R + 0.587G + 0.114B

[0066] The Y component can be obtained through calculation, but since ZYNQ is not good at calculating floating-point numbers and negative numbers, the transformation formula is converted to:

[0067] Y = (77*R + 150*G + 29*B) >> 8

[0068] The `>>` operator performs a right shift operation, which in ZYNQ fills empty spaces with 0s after a right shift. This allows you to convert RGB data to grayscale data in ZYNQ.

[0069] In one embodiment, in the inter-frame differential module, the differential module implements the operation of subtracting the first frame from each frame starting from the second frame of the video, and then outputting a differential video signal. For example... Figure 3 As shown, the frame difference module consists of four parts: write FIFO, RAM, read FIFO, and differential.

[0070] After cross-clock domain processing, the input data is written to RAM via a write FIFO. Starting from the first frame, a counter is used for counting. Starting from the second frame, a separate video timing generation module is used as the read enable for the read FIFO, ensuring that the timing of each frame corresponds to the first frame. Then, differential processing is performed. In the differential operation, the size of two pixel data points is compared, and the smaller data point is subtracted from the larger data point to obtain the differential data of each frame, which is then output to the next module for further processing.

[0071] In one embodiment, one of the functions of the cropping module is to reduce the size of the input image to a suitable size, that is, a power of 2 (N is a natural number). Based on the initial image size, this project crops the image to 256*256.

[0072] After the data is clipped, the IP core used for the subsequent Fourier transform operation needs to convert the data stream into AXI-stream format.

[0073] In one embodiment, in the image Fourier transform module, according to the differential dynamic microscopy method, a Fourier transform is required after the image has undergone difference processing to obtain the spectrum. The Fourier transform operation on the image data is broken down into two steps: performing a Fourier transform on the image row data and performing a Fourier transform on the image column data. Due to the relative independence of row and column transforms, whether the Fourier transform is performed on the row data first or the column data first does not affect the final result. Therefore, in this project, two one-dimensional Fourier transform modules are used to perform Fourier transforms on the row data and column data respectively, thereby realizing the Fourier transform operation on the image.

[0074] In image Fourier transform operations, a transpose module is needed to transpose the image data from the input module, and to schedule the data between two one-dimensional Fourier transform operations, such as... Figure 4 The diagram shown is a structural diagram of the image Fourier transform module.

[0075] like Figure 5 As shown, the transpose operation is implemented as follows: the data stream enters the row Fourier transform IP in the form of row data. After the calculation is completed, the data is stored in DDR. Unlike the general sequential storage method, to facilitate subsequent column-by-column data retrieval, the system stores each row of data sequentially in one column. Data is written by using address transitions to store the row data length. When a row of data is stored, the address is incremented by one and reset to prepare for storing the next row. This method completes the transpose process of the image data.

[0076] like Figure 6 As shown, after the data processed by line transformation is stored in a full frame, it will be stored in a new storage space to avoid mutual interference between read and write operations, thus preventing image tearing. Specifically, an offset address is added to both the write and read addresses. The size of the offset address is the frame number multiplied by the size of a single frame's storage space. Because the read and write speeds remain consistent, allocating three frame numbers is sufficient to avoid image tearing.

[0077] As the second frame of data begins to be stored, the column Fourier transform module also begins to read the stored data. Since the previous data has already undergone image offset processing, the column data can be read directly in sequence. The length of one row is used as the reading length of one burst read, and the reading is divided into 256 reads to complete the reading of the column data and the Fourier transform processing.

[0078] In one embodiment, the radial averaging module mainly consists of a ROM module and a weighted summation module. In the ROM module, the system pre-stores the distances between the four adjacent points of the eight selected points. It's important to note that, according to the bilinear interpolation algorithm, the pixel interval unit in the pixel matrix is ​​always 1, so the distance between the selected point and its four neighboring points is always less than 1. Since the resulting distance value is a floating-point number, and ZYNQ excels at integer operations, a floating-point to integer conversion is necessary. Specifically, the distance weights are multiplied by 256 and then rounded. The calculated result is magnified by a factor of 256 compared to the original result. In digital circuits, division can be performed using bit shifting; shifting the result 8 bits to the right is equivalent to dividing by 256.

[0079] like Figure 7 As shown, when the data stream reaches the selection point where interpolation is needed, the weighted summation module obtains the weights and pixel sizes and temporarily stores them in a register. When enough data is available for the four points, a weighted summation is performed on one selection point, and the result is overwritten in the previous register. After the calculation is completed for the last point, the calculated values ​​of all points are summed and averaged to obtain the required radial average result.

[0080] In summary, this invention proposes a differential dynamic microscopy image processing system and method based on ZYNQ. This system leverages the advantages of ZYNQ's high processing speed and small size to achieve real-time and miniaturized differential dynamic microscopy technology.

Claims

1. A differential dynamic microscopic image processing method based on ZYNQ, characterized in that, Includes the following steps: S1. Acquiring initial microscopic images: The ZYNQ fully programmable on-chip system configures the CMOS industrial camera through the SCCB protocol. The initialized CMOS industrial camera acquires microscopic image information of the sample solution through a microscope. The transmission bit width of the CMOS industrial camera is 8 bits. S2. Grayscale conversion: Because the 16-bit RGB565 output by the camera is sent in two parts, before proceeding to the next step, it is necessary to concatenate the 16-bit RGB565 into 24-bit RGB data, and then remap the 24-bit RGB format data into YCbCr format data to obtain the grayscale pixel information of the Y channel. S3, Inter-frame difference: After the camera is initialized, the first frame image data is taken and stored in RAM. When each frame image from the second frame onwards is passed in, the first frame image data is used to subtract the corresponding pixel data of each frame image from the second frame onwards to obtain the image difference data matrix. S4. Image Fourier Transform: Before performing the Fourier transform, the frame image size is cropped to a power of 2. The image Fourier transform is then split into row Fourier transform and column Fourier transform. After performing a row Fourier transform on the image data, the transformed row data is stored in DDR. When reading, the data is read column by column, and the column data is transformed again. Finally, the Fourier transform result data matrix of the image is output. Based on the row and column independence in the image Fourier transform, one image Fourier transform is split into one row transform and one column transform; the data after converting the video stream format directly enters the row Fourier transform module to perform Fourier transform on the row data, and performs Fourier transform on 256 rows of data in sequence. The calculated data will be stored in DDR, with each row of data stored sequentially in a column; the writing method is that the address jumps according to the length of the row of data, and after completing a row, the address is reset and incremented by one, thus completing the transpose process of the image data; After the data processed by line transformation is full for one frame, it will be stored in a new storage space to avoid mutual interference between read and write data operations, which would cause image tearing. As the second frame of data begins to be stored, the column Fourier transform module also begins to read the stored data. Since the previous data has already undergone image offset processing, the column data can be read directly in sequence. The length of one row is used as the reading length of one burst read, and the column data reading and Fourier transform processing can be completed in 256 reads. S5. Radial averaging: After obtaining the Fourier transform result data matrix, the values ​​of points along a set radius are sampled. Because of isotropy, the sampled values ​​are consistent. The radial average result is obtained by averaging the uniformly distributed points.

2. The differential dynamic microscopic image processing method based on ZYNQ according to claim 1, characterized in that, In step S1, the microscopic image information of the sample solution is transmitted from the microscope to the CMOS industrial camera, and then transmitted to the ZYNQ via an eight-bit data transmission line in DVP video stream format. The camera's shooting settings are configured as follows:

640. 480 pixels, 30 frames per second, RGB565 format.

3. The differential dynamic microscopic image processing method based on ZYNQ according to claim 1, characterized in that, In step S2, since the hardware connection data width of the camera is only eight bits, and the RGB565 format video stream transmission method is to split the RGB data of one pixel into two pixel clocks for transmission, the RGB565 is first split and concatenated into an 8-bit data format to facilitate subsequent grayscale conversion.

4. The differential dynamic microscopic image processing method based on ZYNQ according to claim 1, characterized in that, In step S3, after the camera completes initialization, ZYNQ begins collecting the first frame data from the video stream and stores it in RAM. Once the first frame data is stored and the second frame data begins to be transmitted, each pixel starting from the first pixel of the second frame is subtracted from the corresponding pixel of the first frame stored in RAM. During the subtraction, the two pixels to be subtracted are compared to ensure that the larger number is always subtracted from the smaller number. This results in a frame difference image video stream transmitted in DVP format.

5. The differential dynamic microscopic image processing method based on ZYNQ according to claim 4, characterized in that, In step S4, the DVP format frame difference image video stream obtained in step S3 is cropped in pixel size, from 640 480 pixels, cropped to 256 The resolution is 256 pixels; then the DVP video stream format is converted to AXI-stream format.

6. The differential dynamic microscopic image processing method based on ZYNQ according to claim 1, characterized in that, The specific method to avoid image tearing is to add an offset address to both the write address and the read address. The size of the offset address is the frame sequence number multiplied by the storage space size of a single frame.

7. The differential dynamic microscopic image processing method based on ZYNQ according to claim 1, characterized in that, In step S5, firstly, the single-frame image calculated in step S4 has rotation invariance, that is, with the center of the image as the point, the values ​​within the same radius are the same; eight points that are uniformly distributed and mirror-symmetric within the same radius are selected as the radial average sample values; bilinear interpolation is performed on the selected points, that is, for a selected point, its distance to the four nearest points is used as the weight to multiply by the values ​​corresponding to the four nearest points, thereby obtaining the value of the selected point.

8. A ZYNQ-based differential dynamic microscopic image processing system based on the processing method according to any one of claims 1 to 7, characterized in that, Includes light source, sample solution, microscope, CMOS camera, and ZYNQ image processing module; The light source is mounted on the side of the sample solution, and the emitted light shines on the sample solution. The light is magnified by a microscope, and then a CMOS camera mounted on the microscope acquires the microscopic image. A sensor array is used to detect the light signal magnified by the microscope and convert it into an electrical signal. The microscopic image information is then input into the ZYNQ image processing module that is spliced ​​together with the CMOS camera. The ZYNQ image processing module includes a camera access module based on the SCCB protocol, an AXI-stream video stream format conversion module, a cropping module, a grayscale conversion module, an inter-frame difference module, an image Fourier transform module, and a radial averaging module. The image Fourier transform module includes a transpose scheduling module, a DDR module, and a Fourier transform calculation module; the camera access module based on the SCCB protocol completes the configuration of the camera and video stream access; the AXI-stream video stream format conversion module completes the conversion from DVP video format to AXI-stream format; the cropping module completes the cropping of the image size; the grayscale conversion module obtains the grayscale value in the pixel; the inter-frame difference module completes the difference processing between frame data; the image Fourier transform module completes the Fourier transform of the image; and the radial averaging module completes the interpolation and weighted summation of the sampling points to finally obtain the radial average result.

Images