A pixel misalignment correction method for laser confocal microscope system

Abstract

A pixel misalignment correction method for a laser confocal microscope system includes: image reconstruction based on a galvanometer position signal and a grayscale signal, completing preliminary image reconstruction by filtering the galvanometer position signal and mapping the grayscale signal; pixel misalignment correction in the image reconstruction process, in the image reconstruction process, based on a structural similarity algorithm (SSIM) and a grayscale co-occurrence matrix (GLCM) algorithm, correcting pixel misalignment caused by line break node judgment errors, pixel filling and cropping operations line by line; optimizing the inter-line alignment of the image by jointly using the structural similarity algorithm (SSIM) and the grayscale co-occurrence matrix (GLCM) algorithm, completing pixel misalignment correction. The present invention can solve the pixel misalignment problem generated by the laser confocal microscope system in the image reconstruction process and improve the imaging quality of the laser confocal microscope system.

Description

Technical Field

[0001] The present invention relates to the fields of medical optical imaging, image processing and computer vision technology, and in particular to a pixel misalignment correction method for a laser confocal microscope system. Background Art

[0002] Laser confocal microscopy has a wide range of application value in the fields of biology, medicine, and materials research due to its unique advantages such as high resolution, deep penetration, three-dimensional imaging, high sensitivity, and sample protection. For example, it can be used to observe cell structure, cell membrane fluidity, receptor distribution, organelle structure and distribution changes, etc. It can also be used for drug screening, gene expression research, and the exploration of disease mechanisms. In clinical practice, it can also be used directly to observe lesions in transparent and translucent tissues such as the cornea.

[0003] However, the laser confocal system is different from the traditional imaging method. It adopts a point-by-point imaging solution and cannot directly output a complete two-dimensional image. It can only output one-dimensional signals related to each component in the system. It is necessary to reconstruct the two-dimensional image with the help of various one-dimensional signals and solve the pixel misalignment problem generated during the image reconstruction and signal acquisition process. The specific work content is summarized as follows:

[0004] 1. Convert the signal collected by the laser confocal system into a two-dimensional image: Since the laser confocal system adopts a point-by-point imaging solution, it cannot directly output a complete two-dimensional image. The optical path system used in this case can collect signals including the position signal of the galvanometer, which can reflect the position information of the laser beam on the sample surface; and the current or voltage signal converted by the laser through the pinhole via the photomultiplier tube, which can be converted into a grayscale value to reflect the brightness information at different positions of the sample. The above signals are all one-dimensional signals. In order to achieve the final image reconstruction, the above signals must be processed so that each pixel point is arranged in an orderly manner on the two-dimensional plane, thereby forming a two-dimensional image with spatial and grayscale information.

[0005] 2. Correction of pixel misalignment of two-dimensional images: In the optical path system used in this case. Due to hardware and algorithm problems, pixel misalignment is inevitable. Pixel misalignment is divided into acquisition misalignment and reconstruction misalignment. Acquisition misalignment refers to the incomplete matching of the galvanometer movement and the driving timing of signal acquisition during image acquisition, resulting in a large amount of invalid signals accumulating at the initial end of the one-dimensional data. Such invalid signals cannot be identified in the one-dimensional state and need to be identified and eliminated in the two-dimensional image. Reconstruction misalignment refers to the process of arranging the one-dimensional data output by point-by-point imaging into a two-dimensional image. Due to the error in the judgment of the line break node, as well as the filling and cropping of pixels, the pixels in each row cannot be completely aligned with the previous row. Therefore, if you want to get a readable picture, you need to correct the pixel misalignment of the two-dimensional image.

[0006] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0007] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a pixel misalignment correction method for a laser confocal microscope system.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A pixel misalignment correction method for a laser confocal microscope system comprises the following steps:

[0010] S1. Image reconstruction based on galvanometer position signal and grayscale signal:

[0011] Filtering the galvanometer position signal collected by the laser confocal system, and dividing the image into rows according to the filtered galvanometer position signal;

[0012] Map the grayscale signal into a two-dimensional image and perform preliminary image reconstruction;

[0013] S3. Pixel misalignment correction in image reconstruction:

[0014] In the process of image reconstruction, the pixel misalignment caused by line break node judgment error, pixel filling and cropping operations is corrected row by row based on the structural similarity algorithm (SSIM) and gray level co-occurrence matrix (GLCM) algorithm.

[0015] By jointly using the structural similarity algorithm (SSIM) and the gray-level co-occurrence matrix (GLCM) algorithm, the inter-row alignment of the image is optimized and pixel misalignment correction is completed.

[0016] Furthermore, in step S1, filtering the galvanometer position signal specifically includes:

[0017] A filtering algorithm is used to perform denoising on the galvanometer position signal to obtain a filtered galvanometer position signal;

[0018] According to the filtered galvanometer position signal, a serpentine scanning strategy is used to divide the image rows, in which the galvanometer starts from the upper right corner of the field of view, collects to the left to the leftmost side, then enters the next row and collects to the right again, and this process is repeated until the last row is collected.

[0019] Furthermore, in step S1, dividing the image into rows specifically includes:

[0020] Use a detection window of preset size to determine the line segment based on the compound growth rate of the galvanometer position signal:

[0021] When the compound growth rate exceeds the preset positive growth rate threshold, it is determined to be a right-line paragraph;

[0022] When the compound growth rate is lower than the preset negative growth rate threshold, it is determined to be a left-line paragraph;

[0023] When the compound growth rate is between the preset positive and negative growth rate thresholds, it is determined to be an invalid signal in the non-collection state;

[0024] Invalid signals at the beginning and end of the acquisition are eliminated to obtain valid line segments, and the length of each line is normalized, with the average value of the pixel length of each line rounded up as the standard line length.

[0025] Furthermore, in step S1, mapping the grayscale signal into the two-dimensional image specifically includes:

[0026] Convert the current or voltage signal reflecting the grayscale information into a digital signal and map it to a preset grayscale range;

[0027] Interpolate or evenly crop the grayscale signals of each row according to the standard row length to ensure that the number of pixels in each row is consistent;

[0028] According to the target image width and the number of sampling points N actually collected in each row, the number of sampling points n required for grayscale averaging in each row is calculated, and the grayscale values ​​of n consecutive sampling points in each row are averaged to obtain a single grayscale value to optimize the reconstructed image.

[0029] Furthermore, the method further comprises the following steps:

[0030] S2. Pixel misalignment correction during signal acquisition:

[0031] In the image reconstruction process, the image misalignment evaluation algorithm based on morphological gradient and the single-objective constrained optimization particle swarm algorithm (PSO) are used to determine the optimal misalignment amount;

[0032] The grayscale signal is cropped according to the optimal misalignment amount to eliminate invalid data caused by the asynchrony between the galvanometer movement and signal acquisition, ensuring the correct starting position of image reconstruction;

[0033] Among them, determining the optimal misalignment amount specifically includes:

[0034] Setting a threshold value for the amount of misalignment and constructing images according to different amounts of misalignment;

[0035] Each frame of image is processed based on the morphological gradient algorithm to obtain the image misalignment evaluation coefficient;

[0036] Use the optimization algorithm to search for the minimum value of the image misalignment evaluation coefficient and determine the optimal misalignment amount;

[0037] Apply the optimal misalignment amount to reconstruct the image and complete the image misalignment correction;

[0038] Among them, the image processing based on the morphological gradient algorithm specifically includes:

[0039] Dilate the image and assign the pixel value in the local range to the brightest point.

[0040] Perform an erosion operation on the image and assign the pixel value in the local range to the darkest point;

[0041] Calculate the gray value difference after dilation and erosion as a detection operator;

[0042] Accumulate the detection operators of the entire image to obtain the image misalignment evaluation coefficient;

[0043] Among them, the optimization algorithm is used to search for the minimum value of the image misalignment evaluation coefficient, including:

[0044] Initialize the particle swarm and randomly generate the current displacement and current speed of each particle;

[0045] Determine the evaluation coefficient of each particle according to the dislocation evaluation function corresponding to the current dislocation amount;

[0046] Update the individual optimal value of each particle and the global optimal value of the particle swarm;

[0047] Iteratively update the particle's velocity and displacement according to the inertia weight and acceleration constant;

[0048] When the image misalignment evaluation coefficient reaches the target requirement or the algorithm converges, the optimal misalignment amount corresponding to the global optimal value is output and applied to image reconstruction.

[0049] Furthermore, in step S3, pixel misalignment correction in the image reconstruction process specifically includes:

[0050] According to the texture or structural similarity between rows, pixel misalignment caused by sliding window division and pixel interpolation filling and cropping operations is corrected row by row;

[0051] The structural similarity algorithm (SSIM) and the gray-level co-occurrence matrix algorithm (GLCM) are used together to perform row-by-row correction, where the first n rows are positioned using the GLCM algorithm and the subsequent rows are quickly corrected using the SSIM algorithm.

[0052] Furthermore, in step S3, using the gray level co-occurrence matrix algorithm (GLCM) to locate the first n rows specifically includes:

[0053] Read the original image size and initialize a blank image to store the corrected result;

[0054] Fix the first row in place as a reference row;

[0055] Generate gray-level co-occurrence matrix for the first n rows row by row, and count the gray-level value distribution of pixel pairs;

[0056] Extract texture features based on gray-level co-occurrence matrix and calculate correlation coefficient;

[0057] According to the maximum value of the correlation coefficient, the pixel position of the current row is adjusted to align with the texture distribution of the previous two rows.

[0058] Furthermore, in step S3, using a structural similarity algorithm (SSIM) to quickly correct subsequent rows includes:

[0059] Go to the next line and read the current line data;

[0060] Compare the brightness, contrast and structural similarity of the current row with the previous two rows, and calculate the SSIM evaluation function;

[0061] According to the maximum value of the SSIM evaluation function, adjust the pixel position of the current row to align it with the texture and structure of the previous two rows;

[0062] Repeat the above process until all rows are processed and the corrected image is output.

[0063] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the pixel misalignment correction method for a laser confocal microscope system.

[0064] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the pixel misalignment correction method for a laser confocal microscope system is implemented.

[0065] The present invention has the following beneficial effects:

[0066] The present invention also proposes a pixel misalignment correction method for a laser confocal microscope system. By jointly using a structural similarity algorithm (SSIM) and a gray-level co-occurrence matrix (GLCM) algorithm, the pixel misalignment caused by line break node judgment errors, pixel filling and cropping operations can be efficiently and accurately corrected line by line, significantly optimizing the image inter-row alignment effect, thereby improving the imaging quality of the laser confocal microscope system. The present invention innovatively realizes the effective combination of the structural similarity algorithm and the gray-level co-occurrence matrix, locates the first n rows of the image using the gray-level co-occurrence matrix, and efficiently processes the subsequent rows using the structural similarity algorithm, achieving a new processing method that is superior to the original two in processing efficiency, accuracy, and stability.

[0067] In addition, by combining the image misalignment evaluation algorithm based on morphological gradient and the single-objective constrained optimization particle swarm algorithm (PSO), the optimal misalignment amount can be determined efficiently and accurately, and invalid data caused by the asynchrony between the galvanometer movement and signal acquisition can be effectively eliminated, ensuring the correct starting position of image reconstruction, thereby significantly improving the imaging quality of the laser confocal microscope system. In addition, during the image reconstruction process, this method achieves preliminary reconstruction of the image by filtering the galvanometer position signal and mapping the grayscale signal, further optimizing the image clarity and readability, and providing a reliable basis for subsequent image processing and analysis.

[0068] By considering the acquisition misalignment in the acquisition process caused by the system hardware and the reconstruction misalignment in the image processing link caused by the algorithm, the scope of application is expanded and the defects in the system hardware and software can be effectively compensated.

[0069] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Schematic diagram of the image reconstruction algorithm principle of an embodiment of the present invention.

[0071] Figure 2 The present invention is an embodiment of the present invention, wherein the galvanometer position signal is filtered and divided into rows.

[0072] Figure 3 This is a comparison of the reconstructed image before (left picture) and after (right picture) optimization according to an embodiment of the present invention.

[0073] Figure 4 Collect misaligned images to varying degrees.

[0074] Figure 5 This is a block diagram of an acquisition misalignment correction algorithm according to an embodiment of the present invention.

[0075] Figure 6 This is a verification of the correlation between the misalignment evaluation coefficient and the misalignment amount in an embodiment of the present invention.

[0076] Figure 7 It is a flowchart of a particle swarm optimization algorithm according to an embodiment of the present invention.

[0077] Figure 8 This is a diagram showing the effect of collecting misalignment correction according to an embodiment of the present invention.

[0078] Fig. 9 Reconstruct misaligned images of different degrees.

[0079] Fig.10 This is a flowchart of the SSIM and GLCM pixel misalignment correction algorithm according to an embodiment of the present invention.

[0080] Fig.11 This is the effect of the SSIM pixel misalignment correction algorithm of an embodiment of the present invention.

[0081] Fig.12 This is the effect of the GCLM pixel misalignment correction algorithm according to an embodiment of the present invention.

[0082] Fig.13 4 is a block diagram of the SSIM-GLCM pixel misalignment joint correction algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0084] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0085] The embodiment of the present invention provides a pixel misalignment correction method for a laser confocal microscope system, comprising the following steps:

[0086] Step S1. Image reconstruction based on galvanometer position signal and grayscale signal: filter the galvanometer position signal collected by the laser confocal system, divide the image into rows according to the filtered galvanometer position signal; map the grayscale signal into a two-dimensional image, and perform preliminary image reconstruction.

[0087] In a preferred embodiment, in step S1, filtering the galvanometer position signal specifically includes: using a filtering algorithm to denoise the galvanometer position signal to obtain a filtered galvanometer position signal; according to the filtered galvanometer position signal, using a serpentine scanning strategy to divide the rows of the image, wherein the galvanometer starts from the upper right corner of the field of view, collects to the left to the leftmost side, then enters the next row and collects to the right again, and this process is repeated until the last row is collected.

[0088] In step S1, dividing the rows of the image specifically includes: using a detection window of a preset size (such as 50 pixels), judging the row segment according to the compound growth rate of the galvanometer position signal: when the compound growth rate exceeds the preset positive growth rate threshold (such as +0.2), it is determined to be a right row segment; when the compound growth rate is lower than the preset negative growth rate threshold (such as -0.2), it is determined to be a left row segment; when the compound growth rate is between the preset positive and negative growth rate thresholds (-0.2 to 0.2), it is determined to be an invalid signal in a non-collected state; the invalid signals at the beginning and end of the collection are eliminated to obtain valid row segments, and the length of each row is normalized, and the average value of the pixel length of each row is rounded up as the standard row length.

[0089] In step S1, mapping the grayscale signal to the two-dimensional image specifically includes: converting the current or voltage signal reflecting the grayscale information into a digital signal, and mapping it to a preset grayscale range (such as 0-255); interpolating or evenly cropping the grayscale signals of each row according to the standard row length to ensure that the number of pixels in each row is consistent; calculating the number of sampling points n (such as 0) for grayscale value averaging in each row according to the target image width (such as 250 pixels) and the number of sampling points N actually collected in each row. ), and average the grayscale values ​​of n consecutive sampling points in each row to obtain a single grayscale value to optimize the reconstructed image.

[0090] Step S3. Pixel misalignment correction during image reconstruction: During image reconstruction, pixel misalignment caused by line break node judgment errors, pixel filling and cropping operations is corrected row by row based on the structural similarity algorithm (SSIM) and the gray-level co-occurrence matrix (GLCM) algorithm. By jointly using the structural similarity algorithm (SSIM) and the gray-level co-occurrence matrix (GLCM) algorithm, the inter-row alignment of the image is optimized to complete pixel misalignment correction.

[0091] In a preferred embodiment, in step S3, the pixel misalignment correction in the image reconstruction process specifically includes: correcting the pixel misalignment caused by sliding window division and pixel interpolation filling and cropping operations row by row according to the texture or structural similarity between rows; using the structural similarity algorithm (SSIM) and the gray level co-occurrence matrix algorithm (GLCM) to jointly perform row-by-row correction, wherein the first n rows are positioned using the GLCM algorithm, and the subsequent rows are quickly corrected using the SSIM algorithm.

[0092] In step S3, the gray level co-occurrence matrix algorithm (GLCM) is used to locate the first n rows, specifically including: reading the original image size, initializing a blank image to store the corrected result; fixing the position of the first row as a reference row; generating a gray level co-occurrence matrix for the first n rows row by row, and counting the gray value distribution of pixel pairs; extracting texture features according to the gray level co-occurrence matrix and calculating the correlation coefficient; and adjusting the pixel position of the current row according to the maximum value of the correlation coefficient to align it with the texture distribution of the first two rows.

[0093] In step S3, the structural similarity algorithm (SSIM) is used to quickly correct the subsequent rows, specifically including: entering the next row and reading the current row data; comparing the brightness, contrast and structural similarity of the current row with the previous two rows, and calculating the SSIM evaluation function; according to the maximum value of the SSIM evaluation function, adjusting the pixel position of the current row to align it with the texture and structure of the previous two rows; repeating the above process until all rows are processed and outputting the corrected image.

[0094] In some embodiments, the pixel misalignment correction method for a laser confocal microscope system further includes the following steps:

[0095] Step S2. Pixel misalignment correction during signal acquisition: During the image reconstruction process, the morphological gradient-based image misalignment evaluation algorithm and the single-objective constrained optimization particle swarm algorithm (PSO) are used to determine the optimal misalignment amount; the grayscale signal is cropped according to the optimal misalignment amount to eliminate invalid data caused by the lack of synchronization between the galvanometer movement and signal acquisition, ensuring that the starting position of the image reconstruction is correct.

[0096] In a preferred embodiment, in step S2, determining the optimal misalignment amount specifically includes: setting a threshold value for the change in the misalignment amount, and constructing an image according to different misalignment amounts; processing each frame of the image based on a morphological gradient algorithm to obtain an image misalignment evaluation coefficient; using an optimization algorithm to search for the minimum value of the image misalignment evaluation coefficient to determine the optimal misalignment amount; and applying the optimal misalignment amount to reconstruct the image to complete image misalignment correction.

[0097] In step S2, the image is processed based on the morphological gradient algorithm, specifically including: dilating the image, assigning the pixel value in the local range with the brightest point; eroding the image, assigning the pixel value in the local range with the darkest point; calculating the gray value difference after dilation and erosion as a detection operator; accumulating the detection operators of the entire image to obtain the image misalignment evaluation coefficient.

[0098] In step S2, the optimization algorithm is used to search for the minimum value of the image misalignment evaluation coefficient, including: initializing the particle swarm, randomly generating the current misalignment amount and current speed of each particle; determining the evaluation coefficient of each particle according to the misalignment evaluation function corresponding to the current misalignment amount; updating the individual optimal value of each particle and the global optimal value of the particle swarm; iteratively updating the speed and misalignment amount of the particle according to the inertia weight and the acceleration constant; when the image misalignment evaluation coefficient reaches the target requirement or the algorithm converges, outputting the optimal misalignment amount corresponding to the global optimal value and applying it to image reconstruction.

[0099] The working principle, algorithm example and experimental verification of a specific embodiment of the present invention are further described below.

[0100] First, image reconstruction based on galvanometer position signal and grayscale signal is introduced.

[0101] like Figure 1 As shown in the figure, the galvanometer position signal and grayscale signal that can be collected in the system are two series of numbers with different upper and lower thresholds and different noise levels, but they are mapped one to one. The overall idea of ​​the image reconstruction algorithm is to make a line break judgment based on the characteristics of the galvanometer position signal, and then fill the current or voltage signal reflecting the grayscale information into each row divided according to the galvanometer position signal after normalization processing to complete the preliminary reconstruction of the image.

[0102] Since the galvanometer position signal has a lot of noise, it is necessary to filter the original signal, such as Figure 2 As shown, the gray line is the galvanometer position signal before filtering, and the blue, green, and red lines are the galvanometer position signals after filtering.

[0103] In order to obtain regular galvanometer position signals as much as possible, the galvanometer is controlled to perform a serpentine scanning strategy, that is, starting from the upper right corner, collecting to the left, entering the next row after reaching the leftmost side, and collecting again to the right, and repeating this process until the last row is collected. Therefore, the X galvanometer position signal presents a sawtooth signal during the collection process, with the center of the field of view as the 0 coordinate. A positive slope means that the light source moves to the right, and a negative slope means that the light source moves to the left.

[0104] Since the filtered position signal cannot present a completely straight signal curve when not collected, the algorithm uses 50 pixels as the detection window. When the compound growth rate of 50 pixels exceeds +0.2, it is determined to be a right-line paragraph. When the compound growth rate of 50 pixels is less than -0.2, it is determined to be a left-line paragraph. If the compound growth rate of 50 pixels is between -0.2 and 0.2, it is determined to be an invalid signal in the uncollected state. Since the light source is controlled to move to the upper right corner of the field of view when the collection is started, and the light source is controlled to return to the center from the lower right or lower left corner of the field of view after the collection is completed, the effective signals of the first and last half sections are eliminated, and the result is Figure 2 The division diagram of each row is shown.

[0105] The blue represents the invalid signal that is removed, the red represents the left paragraph, and the green represents the right paragraph. The lines divided by this method will have different lengths, and the lengths of the lines need to be normalized. The average value of the pixel lengths of each line is rounded up as the standard line length.

[0106] For current or voltage signals that reflect grayscale information, the analog signal must first be converted into a digital signal and mapped to a grayscale range of 0 to 255. At this time, the number of pixels originally corresponding to each row is not directly equal to the standard row length. Missing rows need to be interpolated and overlong rows need to be evenly cropped.

[0107] The signal acquisition frequency is significantly higher than the galvanometer movement frequency, resulting in a certain degree of overlap between the collected pixels. Therefore, the reconstructed image shows obvious elongated features. After analyzing the acquisition process, it is concluded that the width of the acquired image should be about 250 pixels, so normalization is performed based on the actual amount of data collected per row N. The grayscale values ​​of the adjacent n (n=⌈N÷250⌉) sampling points are averaged to obtain a single grayscale value, thereby optimizing the reconstructed image.

[0108] Figure 3 Shown is a comparison of the reconstructed image before (left) and after (right) optimization.

[0109] Next, the pixel misalignment correction during the signal acquisition process is described.

[0110] In this case, a dual-galvanometer mechanical galvanometer is used. When the data acquisition instruction and the voltage signal driving the galvanometer are issued at the same time, the voltage signal delay causes the data acquisition to precede the movement of the galvanometer, so that some invalid data is stored at the beginning of the grayscale data. Since the position signal of the galvanometer is used as the reference system during image reconstruction, these invalid data will be mixed in the first row, which will cause the grayscale data corresponding to each subsequent row to be mixed with the grayscale information of the previous row. Figure 4 As shown, based on this principle, images of Demodex mites are used to generate images with different degrees of misalignment for the evaluation of the correction algorithm.

[0111] To correct this type of misalignment, it is necessary to determine the starting position of the valid grayscale data and remove the invalid data in the starting segment.

[0112] The process of collecting misalignment correction algorithm is as follows Figure 5As shown in the figure, first, the threshold of the change of the misalignment amount is set, and the image is constructed according to different misalignment amounts. Each frame of the image is processed using the image misalignment evaluation algorithm based on morphological gradient to obtain the image misalignment evaluation coefficient. If the calculation amount of all the misalignment amounts is too large, the single-objective constraint optimization particle swarm algorithm is combined to search for the minimum value of the image misalignment evaluation coefficient and its corresponding optimal misalignment amount. Finally, the optimal misalignment amount is used to reconstruct the image and complete the image misalignment correction.

[0113] The key to evaluating the quality of the misalignment is to quantify the degree of misalignment. This function is achieved through an image misalignment evaluation algorithm based on morphological gradients. Its core principle is to use graphic expansion and corrosion to detect the degree of pixel mutation.

[0114] Dilation means assigning values ​​to the local area (3×3 pixels in this algorithm) using the brightest point:

[0115] ; (1)

[0116] Erosion is the opposite, all values ​​are assigned using the darkest point:

[0117] ; (2)

[0118] Use the difference in grayscale value after dilation and corrosion as the detection operator:

[0119]

[0120] And the detection operators of the entire image are accumulated as the evaluation coefficient:

[0121]

[0122] Based on the above logic, if the degree of misalignment of the entire image is low, the transition between pixels should be smooth, with few mutations, and a smaller evaluation coefficient will be obtained. On the contrary, if the degree of misalignment is high, there will be many places where the grayscale value changes suddenly, and the evaluation coefficient will be large. Figure 6 As shown, the evaluation coefficient was verified using the generated misalignment image, and it was found that the misalignment amount had a very obvious monotonic positive correlation with the evaluation coefficient. Therefore, it is believed that the algorithm can effectively evaluate the effect of misalignment correction.

[0123] The above algorithm theoretically needs to perform a reconstruction at the beginning of all pixels. When applied to the target 512px*512px scene, it theoretically needs to perform 262144 image reconstructions, which will take up a lot of computing time. Therefore, we consider using a single-objective constrained particle swarm optimization (PSO) algorithm to find the optimal misalignment with fewer iterations. Figure 7 This is the flowchart of the particle swarm optimization algorithm.

[0124] When judging When the target is not reached or convergence is not achieved, the maximum number of iterations should be used. and the inertia weight associated with the current iteration number t and the acceleration constant Update the speed of particle swarm and the amount of displacement :

[0125]

[0126] After testing, when choosing inertia weight , the acceleration constant The highest convergence efficiency can be obtained. Since particle generation is random, the iteration time is also highly random, but the single iteration time is about 0.06-0.08 seconds, and the optimal solution can be obtained after 5-20 iterations. Figure 8 shown.

[0127] Next, the pixel misalignment correction in the image reconstruction process is described.

[0128] Based on the algorithm reconstruction logic above, in the process of distinguishing left-line paragraphs from right-line paragraphs, the use of a 50-pixel sliding window may cause individual pixels at the end of a paragraph to be divided into the next paragraph. In the process of normalizing paragraph length, pixel interpolation, filling, cropping, etc. may also cause pixel misalignment. Fig. 9 As shown, the pixel misalignment caused by these two reasons is random and disordered. It is impossible to use all pixels for overall trial and error, and it can only be corrected row by row based on the texture or structural similarity between rows.

[0129] First, we try to use the Structure Similarity Index Measure (SSIM) and the Gray Level Co-occurrence Matrix (GLCM) algorithm to correct the image row by row. The two algorithms can share the same algorithm framework, and only use different parameters when evaluating the amount of misalignment. Fig.10The flowchart of the SSIM and GLCM pixel misalignment correction algorithm is shown.

[0130] SSIM focuses on the structural similarity of images, which to a certain extent can reflect the human visual system's judgment of the similarity of two images. It is different from the general algorithm that calculates brightness pixel by pixel. The similarity comparison between image x and image y is divided into three dimensions: brightness , contrast and structure Among them, the brightness similarity is mainly evaluated by the grayscale average of the two images. conduct:

[0131]

[0132] The comparison of contrast and structure is mainly based on the covariance of the two images. With standard deviation conduct:

[0133]

[0134] in, All are constants to avoid system errors caused by denominators of 0. The above three dimensions are weighted with constants greater than 0. By weighting, we can get the SSIM evaluation function:

[0135]

[0136] For ease of calculation, it is usually set ,and , SSIM can be simplified to:

[0137]

[0138] In order to test the accuracy, efficiency and stability of the SSIM method for pixel misalignment correction, normal Demodex images were processed with 10 different degrees of misalignment from 1% to 10%, and 100 images were randomly generated at each gradient. The SSIM algorithm was used to process 1,000 images, and the results were as follows: Fig.11 , 12 As shown in the figure. Although SSIM can complete image calibration more accurately, it needs to use the first row as a reference for the entire image. If the first row is misaligned, it will cause the entire image to shift. This means that the SSIM algorithm has a hard time processing the initial row. When the random offset reaches 10%, it is even found that the upper and lower parts of the corrected image are misaligned as a whole.

[0139] From the processing speed point of view, as the amount of misalignment increases, the time required for image processing is also increasing. For the processing efficiency of medical devices, image processing of more than 5 seconds is still too long and needs to be further streamlined. The evaluation coefficient after calibration is generally close to 3370112.00 of the original image, with a difference of less than 1.5%, and the image quality is not significantly different when observed by the naked eye. However, the standard deviation of the 100 images with the same misalignment is too large, which is mainly due to the different degree of overall image offset under the random misalignment of the first row.

[0140] Table 1 Analysis of the efficiency, accuracy and stability of the SSIM pixel misalignment correction algorithm

[0141]

[0142] The GLCM pixel misalignment correction algorithm is consistent with the SSIM method in terms of algorithm framework, except that the index used to evaluate the degree of misalignment is replaced by the GLCM coefficient instead of the SSIM coefficient. The whole process is divided into two steps. The first step is to generate the gray-level co-occurrence matrix, that is, to calculate the gray-level co-occurrence matrix at a specified distance. and direction In the following example, the gray value in the statistical image is The pixel and gray value are The number of times the pixels appear in pairs, and we get a matrix :

[0143] ; (15)

[0144] Among them, when hour, ; when hour, ;when hour, ; when hour, .

[0145] After obtaining the gray-level co-occurrence matrix P, it needs to be normalized:

[0146] ; (16)

[0147] The second step is to extract the texture features of the image based on the gray-level co-occurrence matrix P'. There are many types of texture features that can be obtained based on the gray-level co-occurrence matrix P', including contrast, correlation, energy, homogeneity, etc. Here we are more concerned about the linear relationship between pixels, so we choose correlation as the evaluation indicator:

[0148] ; (17)

[0149] Image correlation can evaluate the degree of image misalignment because when an image is misaligned, the adjacent relationship of pixels is destroyed, the distribution of GLCM changes, and the correlation coefficient decreases. In correction, we want to align the current row with the previous row. Misalignment causes the grayscale symbiosis between the two rows to shift, and the GLCM distribution of adjacent rows should be closer after correction. Using the correlation index, the similarity of the texture distribution of the two rows can be quantified to evaluate the degree of misalignment.

[0150] In the experiment, the algorithm requires a specific row to be compared with the previous n rows. The larger n is, the more accurate the correction is, but the correction time is longer. Through trying multiple n values, it is found that when n increases, the improvement of image correction effect is not obvious, but it will lead to a significant increase in correction time. Through experiments, n=2 is selected as the number of comparison rows selected in the algorithm.

[0151] The same data set is processed using the GLCM algorithm, and the results are as follows Fig.12 As shown in Table 2. Compared with the SSIM method, the time consumption of the GLCM algorithm has increased significantly, almost 3-5 times that of the SSIM method. The calibration accuracy for lower misalignment (1%, 2%) is also slightly lower than that of the SSIM method. The maximum misalignment evaluation coefficient it can achieve is 3415095.30, which is higher than the 3404824.30 that the SSIM method can achieve. With the increase of the misalignment, the calibration ability of the GLCM method has gradually decreased. When the misalignment exceeds 8%, its correction result can be seen with obvious defects by naked eyes. However, the GLCM method also has certain advantages. First of all, it is significantly better than the SSIM method in terms of operational stability, especially in the correction of low misalignment within 4%. In addition, under all misalignment conditions, the GLCM method will not have an overall offset.

[0152] Table 2 Analysis of the efficiency, accuracy and stability of the SSIM pixel misalignment correction algorithm

[0153]

[0154] Combining the characteristics of the two algorithms, we consider using them together, such as Fig.13 As shown in the figure, the GLCM method is first used to correct the first n rows of the image for positioning the entire image. Then the SSIM method is used to correct the remaining rows so that the entire image can be processed more quickly. After experiments, it is determined that when n=10, positioning can be completed accurately without affecting the image processing speed.

[0155] The combination of the two algorithms achieved the effect of 1+1>2. Since the first 10 rows were calculated using the GLCM algorithm, the most time-consuming part of the SSIM algorithm was avoided, which further reduced the calculation time and completed it within 4 seconds. Secondly, the appearance of the positioning row avoided the phenomenon of overall image offset. And when the first 10 rows were determined, the accuracy of the SSIM algorithm was greatly improved. For 1,000 images in the same database, the combined algorithm completely and correctly corrected all images back to their initial state.

[0156] Table 3 Analysis of the efficiency, accuracy and stability of the SSIM-GLCM pixel misalignment joint correction algorithm

[0157]

[0158] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0159] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0160] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0161] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0162] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0163] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0164] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0165] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0166] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0167] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0168] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0169] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0170] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A pixel misalignment correction method for a laser confocal microscope system, characterized in that: The following steps are involved: S1. Image reconstruction based on galvanometer position signal and grayscale signal: Filtering the galvanometer position signal collected by the laser confocal system, and dividing the image into rows according to the filtered galvanometer position signal; Map the grayscale signal to a two-dimensional image to perform preliminary image reconstruction; convert the current or voltage signal reflecting the grayscale information into a digital signal and map it to a preset grayscale range; interpolate or evenly crop the grayscale signals of each row according to the standard row length to ensure that the number of pixels in each row is consistent; calculate the number of sampling points n for grayscale value averaging in each row based on the target image width and the number of sampling points N actually collected in each row, and average the grayscale values ​​of the consecutive n sampling points in each row to obtain a single grayscale value to optimize the reconstructed image; S3. Pixel misalignment correction in image reconstruction: In the process of image reconstruction, the pixel misalignment caused by line break node judgment error, pixel filling and cropping operations is corrected row by row based on the structural similarity algorithm (SSIM) and gray level co-occurrence matrix (GLCM) algorithm. By jointly using the structural similarity algorithm (SSIM) and the gray-level co-occurrence matrix (GLCM) algorithm, the inter-row alignment of the image is optimized and pixel misalignment correction is completed.

2. The pixel misalignment correction method for a laser confocal microscope system according to claim 1, characterized in that: In step S1, filtering the galvanometer position signal specifically includes: A filtering algorithm is used to perform denoising on the galvanometer position signal to obtain a filtered galvanometer position signal; According to the filtered galvanometer position signal, a serpentine scanning strategy is used to divide the image rows, in which the galvanometer starts from the upper right corner of the field of view, collects to the left to the leftmost side, then enters the next row and collects to the right again, and this process is repeated until the last row is collected.

3. The pixel misalignment correction method for a laser confocal microscope system according to claim 1, characterized in that: In step S1, dividing the image into rows specifically includes: Use a detection window of preset size to determine the line segment based on the compound growth rate of the galvanometer position signal: When the compound growth rate exceeds the preset positive growth rate threshold, it is determined to be a right-line paragraph; When the compound growth rate is lower than the preset negative growth rate threshold, it is determined to be a left-line paragraph; When the compound growth rate is between the preset positive and negative growth rate thresholds, it is determined to be an invalid signal in the non-collection state; Invalid signals at the beginning and end of the acquisition are eliminated to obtain valid line segments, and the length of each line is normalized, with the average value of the pixel length of each line rounded up as the standard line length.

4. The pixel misalignment correction method for a laser confocal microscope system according to any one of claims 1 to 3, characterized in that: The following steps are also included: S2. Pixel misalignment correction during signal acquisition: In the image reconstruction process, the image misalignment evaluation algorithm based on morphological gradient and the single-objective constrained optimization particle swarm algorithm (PSO) are used to determine the optimal misalignment amount; The grayscale signal is cropped according to the optimal misalignment amount to eliminate invalid data caused by the asynchrony between the galvanometer movement and signal acquisition, ensuring the correct starting position of image reconstruction; Among them, determining the optimal misalignment amount specifically includes: Setting a threshold value for the amount of misalignment and constructing images according to different amounts of misalignment; Each frame of image is processed based on the morphological gradient algorithm to obtain the image misalignment evaluation coefficient; Use the optimization algorithm to search for the minimum value of the image misalignment evaluation coefficient and determine the optimal misalignment amount; Apply the optimal misalignment amount to reconstruct the image and complete the image misalignment correction; Among them, the image processing based on the morphological gradient algorithm specifically includes: Dilate the image and assign the pixel value in the local range to the brightest point. Perform an erosion operation on the image and assign the pixel value in the local range to the darkest point; Calculate the gray value difference after dilation and erosion as a detection operator; Accumulate the detection operators of the entire image to obtain the image misalignment evaluation coefficient; Among them, the optimization algorithm is used to search for the minimum value of the image misalignment evaluation coefficient, including: Initialize the particle swarm and randomly generate the current displacement and current speed of each particle; Determine the evaluation coefficient of each particle according to the dislocation evaluation function corresponding to the current dislocation amount; Update the individual optimal value of each particle and the global optimal value of the particle swarm; Iteratively update the particle's velocity and displacement according to the inertia weight and acceleration constant; When the image misalignment evaluation coefficient reaches the target requirement or the algorithm converges, the optimal misalignment amount corresponding to the global optimal value is output and applied to image reconstruction.

5. The pixel misalignment correction method for a laser confocal microscope system according to any one of claims 1 to 3, characterized in that: In step S3, pixel misalignment correction in the image reconstruction process specifically includes: According to the texture or structural similarity between rows, pixel misalignment caused by sliding window division and pixel interpolation filling and cropping operations is corrected row by row; The structural similarity algorithm (SSIM) and the gray-level co-occurrence matrix algorithm (GLCM) are used together to perform row-by-row correction, where the first n rows are positioned using the GLCM algorithm and the subsequent rows are quickly corrected using the SSIM algorithm.

6. The pixel misalignment correction method for a laser confocal microscope system according to any one of claims 1 to 3, characterized in that: In step S3, the gray level co-occurrence matrix algorithm (GLCM) is used to locate the first n rows, specifically including: Read the original image size and initialize a blank image to store the corrected result; Fix the first row in place as a reference row; Generate gray-level co-occurrence matrix for the first n rows row by row, and count the gray-level value distribution of pixel pairs; Extract texture features based on gray-level co-occurrence matrix and calculate correlation coefficient; According to the maximum value of the correlation coefficient, the pixel position of the current row is adjusted to align with the texture distribution of the previous two rows.

7. The pixel misalignment correction method for a laser confocal microscope system according to any one of claims 1 to 3, characterized in that: In step S3, the structural similarity algorithm (SSIM) is used to quickly correct the subsequent rows, including: Go to the next line and read the current line data; Compare the brightness, contrast and structural similarity of the current row with the previous two rows, and calculate the SSIM evaluation function; According to the maximum value of the SSIM evaluation function, adjust the pixel position of the current row to align it with the texture and structure of the previous two rows; Repeat the above process until all rows are processed and the corrected image is output.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the pixel misalignment correction method for a laser confocal microscope system as described in any one of claims 1 to 7 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the pixel misalignment correction method for a laser confocal microscope system as described in any one of claims 1 to 7 is implemented.

Images