Image Processing Apparatus and Method Based on Images Acquired by a Fluorescence Microscope

Abstract

The present application discloses an image processing device and method based on fluorescence microscope acquisition. The method includes: obtaining a to-be-processed image acquired by a fluorescence microscope and generating an acquisition environment distribution feature vector of the to-be-processed image; inputting the to-be-processed image and its acquisition environment distribution feature vector into a pre-trained multi-class image processing model to output a target image corresponding to the to-be-processed image. Among them, the pre-trained multi-class image processing model includes: an image attention distribution module, an image distributed processing module, and an image feature fusion output module. The image distributed processing module includes several types of image processing models. Thus, differential image processing for to-be-processed images under different acquisition conditions is realized, and the adaptability and processing effect of the image processing model are improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and bioengineering technologies, and particularly to an image processing device and method based on fluorescence microscope acquisition. Background Art

[0002] Deoxyribonucleic acid (DNA) microarray, that is, gene chip, is a kind of application of microarray technology, used for high-throughput qualitative or quantitative measurement of nucleic acids existing in organisms. Fluorescence microscopy imaging is the main method for gene chip data acquisition. However, the currently acquired gene chip images have the problem of low image quality. The main reasons include:

[0003] The types and doses of fluorescence need to be strictly controlled to maintain the biological activity of DNA, but the fluorescence interference excited in adjacent regions on the same focal plane will reduce the contrast of the acquired image; the errors accompanied in the optical acquisition and imaging processes, such as the working errors of electronic devices and signal conversion losses, result in image noise; the defects of the LED light source in the upright fluorescence microscope and its susceptibility to axial and lateral interference lead to uneven illumination, reducing the proportion of effective information. Currently, although a laser confocal system can obtain high-quality and high-resolution gene chip images, the equipment cost is high.

[0004] The Chinese invention patent with the patent application number 202310631275.4 discloses a method for model training and an image processing method and device, which use the gene chip images acquired by a fluorescence microscope as sample images, and use the loss value determined by the signal-to-noise ratio of the sample images and the processed images output by the model to train the image processing model, and apply the trained image processing model in practice to obtain high-quality and high-resolution gene chip images.

[0005] However, in actual application scenarios, due to the complex differences in the acquisition conditions of fluorescence microscopy imaging (such as fluorescence dye types, light intensities, experimental conditions, etc.), different types of sample images may have different characteristics and processing requirements. A fixed and unified image processing model is difficult to adapt to the sample images under all acquisition conditions. Under the condition of optimizing the global training parameters that pay attention to all acquisition conditions, it may perform poorly in some local and single acquisition conditions, lacking a targeted image processing process. Summary of the Invention

[0006] The present application provides an image processing method based on fluorescence microscope acquisition, which realizes differential image processing for the to-be-processed images under different acquisition conditions, and improves the adaptability and processing effect of the image processing model.

[0007] The present application provides an image processing method based on fluorescence microscope acquisition, including:

[0008] S101. Obtain the image to be processed collected by the fluorescence microscope and generate the acquisition environment distribution feature vector of the image to be processed.

[0009] S102. Input the image to be processed and its acquisition environment distribution feature vector into the pre-trained multi-class image processing model, and output the target image corresponding to the image to be processed.

[0010] Among them, the pre-trained multi-class image processing model includes: an image attention distribution module, an image distributed processing module, and an image feature fusion output module. The image distributed processing module includes several types of image processing models. The multi-class image processing model is used for:

[0011] A1. Transmit the received image to be processed and its acquisition environment distribution feature vector to the image attention distribution module, generate the distributed processing value sequence corresponding to the image to be processed, and transmit it to the image distributed processing module.

[0012] A2. According to the received distributed processing value sequence, the image distributed processing module inputs the image to be processed into the corresponding type of image processing model respectively, outputs several types of target images, and transmits them to the image feature fusion output module.

[0013] A3. The image feature fusion output module performs weighted fusion on the received several types of target images according to the distributed processing value sequence and outputs the final target image.

[0014] Preferably, the image distributed processing module includes several types of image processing models, and each type of image processing model is assigned a unique image class label, which is determined according to the center point of the acquisition environment distribution feature vector of the training image set in the corresponding type of image processing model.

[0015] Preferably, the training process of the several types of image processing models specifically includes:

[0016] C1. Collect a large number of sample images collected by the fluorescence microscope and generate the acquisition environment distribution feature vector of the sample images.

[0017] C2. Input all the collected acquisition environment distribution feature vectors into the pre-set clustering algorithm to obtain k clusters, and form a training image set with the sample images corresponding to all the acquisition environment distribution feature vectors in each cluster, that is, each cluster corresponds to a training image set.

[0018] C3. Construct k types of image processing models with a model structure composed of several feature extraction layers and several transposed convolutional layers as the image processing models to be trained.

[0019] C4. Based on each training image set, input the sample images therein into the corresponding image processing model to be trained, and output class target images;

[0020] C5. Based on the class target images output by each image processing model and the sample images input thereto, use a pre-set model optimization index determination algorithm to generate target evaluation indexes;

[0021] C6. Use the target evaluation indexes to train the image processing models. Repeat and loop to execute steps C4 to C6, continuously iterate and train, and gradually reach the target state of the target evaluation indexes.

[0022] Preferably, the image attention distribution module is specifically used for:

[0023] B1. Obtain the image class labels of all image processing models in the image distributed processing module;

[0024] B2. Calculate the similarity values between the acquisition environment distribution feature vectors of the images to be processed and all image class labels respectively;

[0025] B3. Sequentially form the distributed processing value sequences with all the calculated similarity values. Each similarity value in the distributed processing value sequence corresponds to the corresponding image processing model respectively.

[0026] Preferably, the pre-set model optimization index determination algorithm specifically includes:

[0027] S201. Divide the class target images and the sample images input thereto into n sub-image blocks respectively, and calculate the feature vectors of each sub-image block. The sub-image block feature vector of the sample image is , and the sub-image block feature vector of the class target image is corresponds one by one to ;

[0028] S202. Calculate the mutual information of the corresponding sub-image blocks in the sample image and the class target image in terms of the feature vectors, and calculate the visual information fidelity of each sub-image block in the class target image relative to the sample image according to the following formula:

[0029]

[0030] Among them, is the visual information fidelity, is the feature vector of the i-th sub-image block of the sample image, is the feature vector of the i-th sub-image block of the class target image, n is the number of sub-image blocks divided by the sample image and the class target image respectively, is and the mutual information between is the mutual information with itself;

[0031] S203. Calculate the pixel difference degree between each sub-image block in the class target image and the corresponding sub-image block in the sample image;

[0032] S204. Determine the ratio of the visual information fidelity to the pixel difference degree as the covariant direction value, and combine the visual information fidelity to obtain the target evaluation index.

[0033] Preferably, the target evaluation index is calculated according to the following formula:

[0034]

[0035]

[0036] where L is the target evaluation index, n is the number of sub-image blocks into which the class target image is divided, is the image quality index of the i-th sub-image block in the class target image, is the visual information fidelity of the i-th sub-image block in the class target image, is the covariant direction value of the i-th sub-image block in the class target image, is a preset weight parameter used to adjust the influence degree of the covariant direction value on the image quality index, is the influence degree value of the image quality index of the i-th sub-image block on the target evaluation index.

[0037] Preferably, in the S201, the method for dividing sub-image blocks is generated according to a preset adaptive segmentation mechanism, and the adaptive segmentation mechanism specifically includes:

[0038] S301. Perform connected component recognition on the sample image to obtain several connected components;

[0039] S302. Based on each connected component in the sample image, use the pixel coordinates of its central pixel point in the sample image as the position coordinates of the connected component, and calculate the distance values between every two connected components respectively;

[0040] S303. Based on the distance values between every two connected components, use a preset clustering algorithm to obtain several clusters, each cluster includes at least one connected component, and use the maximum circumscribed rectangle of each cluster as the key area;

[0041] S304. Based on all the key areas recognized in the sample image, use a non-uniform division strategy to divide the sample image into several sub-image blocks, ensuring that each sub-image block includes at most one key area;

[0042] S305. Divide sub - image blocks in the class target image corresponding to the positions of all sub - image blocks of the sample image, which is consistent with the sample image.

[0043] Preferably, the influence degree value of the image quality index of the i - th sub - image block on the target evaluation index is calculated according to the following formula:

[0044]

[0045] Where, is the influence degree value of the image quality index of the i - th sub - image block on the target evaluation index, is the area value of the i - th sub - image block, S is the area of the sample image, is the number of connected components in the i - th sub - image block, N is the total number of connected components in the sample image, and are preset weight factors for the importance of the area value and the number of connected components to the influence degree value, .

[0046] Preferably, after S102, the method further includes:

[0047] S103. Use the adaptive segmentation mechanism to divide the image to be processed into several sub - image blocks and assign a unique position label to each sub - image block;

[0048] S104. Based on the position labels of the sub - image blocks of the image to be processed, determine the sub - image blocks with the same position labels in its target image, and calculate the maximum area values of the connected components in the sub - image blocks with this position label in the image to be processed and the target image respectively, obtaining a first area value and a second area value;

[0049] S105. Based on each sub - image block in the target image, calculate the difference between its image quality index and the maximum area domain, and determine whether there are abnormal sub - image blocks;

[0050] S106. If there are, segment the abnormal sub - image blocks from the image to be processed to obtain sub - samples, execute step S102, input them into the multi - class image processing model, output the sub - target images corresponding to the sub - samples, and according to their position labels, replace the sub - image blocks at these position labels in the target image to obtain a new target image, replacing the original target image.

[0051] The present application also provides an image processing device based on fluorescence microscope acquisition, including: an acquisition module and an image processing module. The image processing module includes multiple types of image processing models, and the multiple types of image processing models include: an image attention distribution module, an image distributed processing module, and an image feature fusion output module. The acquisition module is configured to: obtain a to-be-processed image acquired by a fluorescence microscope, generate an acquisition environment distribution feature vector of the to-be-processed image, and input it into the image processing module. The image processing module is configured to: input the to-be-processed image and its acquisition environment distribution feature vector into the pre-trained multiple types of image processing models, and output a target image corresponding to the to-be-processed image.

[0052] The multiple types of image processing models are configured to: A1. Transmit the received to-be-processed image and its acquisition environment distribution feature vector to the image attention distribution module, generate a distributed processing value sequence corresponding to the to-be-processed image, and transmit it to the image distributed processing module; A2. The image distributed processing module, according to the received distributed processing value sequence, inputs the to-be-processed image into the corresponding type of image processing model respectively, outputs several types of target images, and transmits them to the image feature fusion output module; A3. The image feature fusion output module performs weighted fusion on the received several types of target images according to the distributed processing value sequence, and outputs the final target image.

[0053] One or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0054] By introducing the acquisition environment distribution feature vector and multiple types of image processing models, differential image processing for to-be-processed images under different acquisition conditions is realized based on the distributed processing value sequence, improving the adaptability and effect of image processing; solving the problem that a fixed and unified image processing model is difficult to adapt to sample images under all acquisition conditions, and improving the flexibility and accuracy of image processing;

[0055] The image quality index combines two important indexes, VIF and pixel difference degree, and can more comprehensively reflect the quality of image processing; the image quality index clearly quantifies the change directionality between the visual information fidelity and the pixel difference degree by introducing the covariant direction value, making the evaluation result more instructive; by adjusting the weight parameter α, the evaluation focus of CQI can be customized according to specific application scenarios and requirements. Based on the joint evaluation of the visual information fidelity and the covariant direction value, the contingency caused by a single index judgment can be avoided, and the visual information fidelity is constrained by the change directionality, making the evaluation result more reliable;

[0056] Through the adaptive segmentation mechanism, the key information features in the fluorescence microscope images can be captured more accurately. During the training process of the image processing model, the key regions of the sample images are analyzed differentially and mapped to the class target images, enabling a more accurate and refined target evaluation index from local to global analysis, reflecting the processing effect of the model on the images, and improving the flexibility and accuracy of image processing, especially for images containing complex biological structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic flow chart of the image processing method based on fluorescence microscope acquisition according to an embodiment of the present invention;

[0058] Figure 2 Block diagram of the structure of the image processing device based on fluorescence microscope acquisition according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] To facilitate the understanding of the present invention, the present application will be described more comprehensively with reference to the relevant drawings; the preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0060] It should be noted that the terms "vertical", "horizontal", "upper", "lower", "left", "right" and similar expressions used herein are for illustrative purposes only and do not represent the only embodiments.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs; the terms used in the description of the present invention in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0062] Embodiment 1: Figure 1 Schematic flow chart of the image processing method based on fluorescence microscope acquisition according to an embodiment of the present invention.

[0063] As Figure 1 shown, an image processing method based on fluorescence microscope acquisition includes the following steps:

[0064] S101, obtaining the image to be processed collected by the fluorescence microscope and generating the acquisition environment distribution feature vector of the image to be processed.

[0065] Among them, the acquisition environment distribution feature vector includes illumination conditions, fluorescence dye types, and environmental temperature.

[0066] In a fluorescence microscope, the wavelength, intensity of the excitation light, and the indoor light conditions all have a direct impact on the excitation effect of the fluorescent dye. Too strong light may cause fluorescence quenching, while too weak light may not be able to effectively excite the fluorescent dye, affecting the image quality; the selection of the fluorescent dye is crucial for specific biological samples or cell structures. Different dyes may have different affinities for specific cell components. Using an inappropriate dye may lead to problems such as weak signals, high background noise, or poor specificity; the environmental temperature has a certain impact on the performance of the fluorescence microscope and the stability of the sample. An appropriate environmental temperature can maintain the stability of the microscope and the activity of the sample. Too high or too low temperature may cause the performance of the microscope to decline or the sample to deform, thus affecting the image quality.

[0067] Specifically, in the field of fluorescence microscopy imaging, due to the complex differences in acquisition conditions, different types of sample images may have different characteristics and processing requirements. Collect the sample images acquired by the fluorescence microscope and generate corresponding acquisition environment distribution feature vectors according to the acquisition conditions.

[0068] S102, Input the image to be processed and its acquisition environment distribution feature vector into a pre-trained multi-class image processing model, and output the target image corresponding to the image to be processed.

[0069] Thus, with the support of the multi-class image processing model, it is possible to output the optimal processing results for different types of images to be processed and acquisition conditions, improving the usability and accuracy of the images.

[0070] In some embodiments, the pre-trained multi-class image processing model includes: an image attention distribution module, an image distributed processing module, and an image feature fusion output module, and connections are established between the modules in sequence.

[0071] Specifically, the multi-class image processing model is used for:

[0072] A1. Transmit the received image to be processed and its acquisition environment distribution feature vector to the image attention distribution module, generate a distributed processing value sequence corresponding to the image to be processed, and transmit it to the image distributed processing module;

[0073] Among them, the image attention distribution module is specifically used for:

[0074] B1. Obtain the image class labels of all class image processing models in the image distributed processing module;

[0075] B2. Calculate the similarity values between the acquisition environment distribution feature vector of the image to be processed and all image class labels respectively (the Euclidean distance algorithm can be used for calculation, and the present invention will not elaborate on this);

[0076] B3. Arrange all the calculated similarity values in sequence to form a distributed processing value sequence. Each similarity value in the distributed processing value sequence corresponds to a corresponding class of image processing models respectively.

[0077] A2. According to the received distributed processing value sequence, the image distributed processing module inputs the images to be processed into the corresponding class of image processing models respectively, outputs several classes of target images, and transmits them to the image feature fusion output module.

[0078] Among them, the image distributed processing module includes several classes of image processing models. Each class of image processing models is assigned a unique image class label, and the image class label is determined according to the center point of the acquisition environment distribution feature vector of the training image set in the corresponding class of image processing models.

[0079] Among them, each class of image processing models includes a feature extraction layer and a transposed convolution layer. Each feature extraction layer is used to perform convolution processing on the image features output by the previous network layer to obtain the convolved image features, and perform feature padding to restore the image feature size. Each transposed convolution layer is used to process the image features output by the previous network layer and expand the feature size.

[0080] It should be noted that for the specific processing process principles of the feature extraction layer and the transposed convolution layer in the model, please refer to the relevant existing technologies, and the present invention will not elaborate on this.

[0081] Among them, the training process of several classes of image processing models specifically includes:

[0082] C1. Collect a large number of sample images collected by a fluorescence microscope and generate the acquisition environment distribution feature vectors of the sample images.

[0083] C2. Input all the collected acquisition environment distribution feature vectors into a pre-set clustering algorithm (K-means clustering) to obtain k clusters. Combine the sample images corresponding to all the acquisition environment distribution feature vectors in each cluster into a training image set, that is, each cluster corresponds to a training image set. For the detailed clustering process principle, please refer to the relevant existing technologies, and the present invention will not elaborate on this. Use the center point of each cluster as the image class label of the corresponding training image set.

[0084] C3. Construct k classes of image processing models with a model structure composed of several feature extraction layers and several transposed convolution layers as the image processing models to be trained.

[0085] C4. Based on each training image set, input the sample images in it into the corresponding image processing model to be trained and output class target images.

[0086] C5. Based on the class target images output by each class image processing model and their input sample images, use a pre-set model optimization index determination algorithm to generate target evaluation indicators.

[0087] C6. Use the target evaluation indicators to train the class image processing model (gradient descent and other optimization algorithms can be used, referring to relevant existing technologies, which will not be elaborated in this invention). Repeat steps C4 to C6 in a loop, continuously iterate and train, and gradually reach the target state of the target evaluation indicators (the larger the target evaluation indicators, the better the model performance), so that the model output is closer to the expected output.

[0088] Thus, collect a large number of sample images, generate acquisition environment distribution feature vectors; use the clustering algorithm to obtain k clusters, and each cluster corresponds to a training image set; construct k class image processing models; train the models based on the training image sets, and continuously optimize the model parameters until the target state of the target evaluation indicators is reached; provide well-trained class image processing models for subsequent image processing; obtain k well-trained class image processing models, which can process sample images under different acquisition conditions.

[0089] A3. The image feature fusion output module performs weighted fusion on the received several class target images according to the distributed processing value sequence, and outputs the final target image.

[0090] Specifically, each pixel coordinate in the class target image corresponds to a pixel gray value. The gray values of each pixel coordinate in several class target images are weighted and summed to obtain the target gray value of the target image at this pixel coordinate; among them, the weight value corresponding to each class target image is the similarity value of this class target image in the distributed processing value sequence.

[0091] The technical solutions in the embodiments of the present application above have at least the following technical effects or advantages:

[0092] By introducing the acquisition environment distribution feature vector and multi-class image processing models, differential image processing is realized for the to-be-processed images under different acquisition conditions based on the distributed processing value sequence, improving the adaptability and effect of image processing; solving the problem that a fixed and unified image processing model is difficult to adapt to the sample images under all acquisition conditions, and improving the flexibility and accuracy of image processing.

[0093] Embodiment 2: Improve and define the model optimization index determination algorithm in Embodiment 1 to optimize the performance of the class image processing model.

[0094] Therefore, the embodiments of the present application are optimized to a certain extent on the basis of the above embodiments.

[0095] In some embodiments, the pre-set model optimization metric determination algorithm specifically includes:

[0096] S201: Divide the class target image and its input sample image into n sub-image blocks respectively (the sub-image blocks with the same position coordinates in the class target image and its sample image present a one-to-one relationship), and calculate the feature vector of each sub-image block. Specifically: for the sample image and the class target image and correspond one by one;

[0097] Specifically, the way to obtain the feature vector of the sub-image block is as follows:

[0098] Extract through a pre-set convolutional neural network (CNN). In the CNN, the sub-image block is input into the network, and after multiple convolutional and pooling operations, a fixed-length feature vector is finally obtained. These feature vectors capture the high-level visual features of the sub-image block, specifically including: preprocessing the sub-image block, such as normalization, denoising, etc.; performing a convolutional operation on the image block through the convolutional layer to extract local features; performing downsampling on the convolutional features through the pooling layer to reduce the feature dimension; inputting the pooled features into the fully connected layer to obtain a fixed-length feature vector; in the CNN, the parameter indicators include the convolutional kernel size, pooling window size, number of network layers, activation function, etc. These parameters will affect the extraction effect and calculation efficiency of the feature vector, and are set according to the actual scenario requirements in the actual application process. The present invention will not elaborate on this.

[0099] S202: Calculate the mutual information between the corresponding sub-image blocks in the sample image and the class target image, and calculate the visual information fidelity of each sub-image block in the class target image relative to the sample image according to the following formula:

[0100]

[0101] where is the visual information fidelity, is the feature vector of the i-th sub-image block of the sample image, is the feature vector of the i-th sub-image block of the class target image, n is the number of sub-image blocks divided for the sample image and the class target image respectively, is and the mutual information between, is its own mutual information (as a normalization factor).

[0102] VIF is used to evaluate image quality. It measures the image quality loss by comparing the visual information fidelity between the sample image and the class target image. Mutual information is a measure of the degree of mutual dependence between two random variables. In VIF, mutual information is used to measure the similarity between the sub-image block Ri of the sample image and the sub-image block Di of the class target image. It should be noted that the calculation of mutual information usually involves the similarity measurement between the feature vectors of the sub-image blocks. The feature vectors can be extracted by deep learning methods such as convolutional neural network (CNN). The specific calculation formula of mutual information may vary depending on the implementation method, but the general form is: MI( , ) = H( )+H( )-H( , ), where H( ) and H( ) are the entropies of and respectively, and H( , ) is the joint entropy of and . In practical applications, the entropy and joint entropy can be calculated by estimating the probability distribution of the sub-image block feature vectors. Calculation of entropy: The calculation of entropy usually involves the estimation of the probability distribution. In image processing, the entropy can be calculated by estimating the probability distribution of the image block feature vectors, which is usually achieved by methods such as histogram statistics or kernel density estimation; Calculation of joint entropy: The calculation of joint entropy is similar, but the joint probability distribution of the two image block feature vectors needs to be considered. The calculation principle of entropy can be understood by referring to the relevant existing technologies, and the present invention will not elaborate on this.

[0103] S203. Based on each sub-image block in the class target image, calculate the pixel difference degree between it and the corresponding sub-image block in the sample image (take the variance value of the pixel differences corresponding to all pixel grids in the sub-image block);

[0104] S204. Determine the ratio of the visual information fidelity to the pixel difference degree as the covariance direction value, and calculate the target evaluation index according to the following formula:

[0105]

[0106]

[0107] where L is the target evaluation index, n is the number of sub-image blocks divided in the class target image, is the image quality index of the i-th sub-image block in the class target image, is the visual information fidelity of the i-th sub-image block in the class target image, is the covariant direction value of the $i$-th sub-image block in the class target image, is a preset weight parameter used to adjust the influence degree of the covariant direction value in the image quality index, is the influence degree value of the image quality index of the $i$-th sub-image block on the target evaluation index. Thus, the greater the visual information fidelity and the greater the covariant direction value, the higher the image quality. Among them, the covariant direction value is used to quantify the negative correlation between the visual information fidelity and the pixel difference degree, and more directly reflects the directionality that "the greater the pixel error degree, the smaller the VIF".

[0108] In some embodiments, in step C6, the training model is continuously iterated to gradually reach the target state of the target evaluation index. The target state is preset and can be set according to the actual situation, and is used to represent the state where the model's ability to process images meets the standard. When the target evaluation index stably reaches the target state during the training process, the training of the model is completed.

[0109] The technical solutions in the above embodiments of the present application at least have the following technical effects or advantages:

[0110] The image quality index combines two important indexes, VIF and pixel difference degree, and can more comprehensively reflect the quality of image processing; by introducing the covariant direction value, the image quality index clearly quantifies the change directionality between the visual information fidelity and the pixel difference degree, making the evaluation result more instructive; by adjusting the weight parameter $\alpha$, the evaluation focus of CQI can be customized according to specific application scenarios and requirements. Based on the joint evaluation of the visual information fidelity and the covariant direction value, the contingency caused by a single index judgment can be avoided, and the visual information fidelity is constrained by the change directionality, making the evaluation result more reliable;

[0111] The target evaluation index is generated according to the CQI of each sub-image block in the class target image. The larger the target evaluation index, the better the image processing ability of the class image processing model, which can more accurately reflect the performance of the model in the image processing process, especially in the balance between the processing effect and the pixel-level accuracy, and can also more clearly guide the training optimization direction of the class image processing model, making the image quality processing reach a more ideal state;

[0112] When optimizing parameters during the model training process, the analysis of the training image before and after is converted from local sub-image block analysis to global image quality analysis, combined with local differential fusion, so that the final global evaluation index can better reflect the overall quality of the image, improve the image processing effect of the model, and avoid the omission of the analysis results of some pixel regions caused by global image analysis.

[0113] Embodiment 3: In Embodiment 2, the sub-image blocks of the model optimization index determination algorithm are usually obtained based on uniform partitioning. However, the images collected by fluorescence microscopes often contain specific biological structures or other contents, such as DNA, which have higher brightness or fluorescence intensity in the images. Uniform partitioning may not effectively capture the features of these key information, resulting in the neglect of the special processing requirements of these important regions in model training during image quality assessment.

[0114] Therefore, the embodiment of the present application is optimized to a certain extent based on the above embodiment.

[0115] In some embodiments, in step S201, the method for partitioning sub-image blocks can be generated according to a preset adaptive segmentation mechanism. The adaptive segmentation mechanism specifically includes:

[0116] S301, Perform connected component recognition on the sample image to obtain several connected components.

[0117] Specifically, based on the gray values of all pixel grids in the sample image, use threshold segmentation or edge detection methods to identify the connected components in the sample image. The connected components usually correspond to regions with higher fluorescence intensity, such as DNA distribution regions.

[0118] S302, Based on each connected component in the sample image, use the pixel coordinates of its central pixel point in the sample image as the position coordinates of the connected component, and calculate the distance values between every two connected components respectively.

[0119] S303, Based on the distance values between every two connected components, use a preset clustering algorithm (for example, K-means clustering) to obtain several clusters (assumed to be r), each cluster includes at least one connected component, and use the maximum circumscribed rectangle of each cluster as the key region.

[0120] S304, Based on all the key regions identified in the sample image, use a non-uniform partitioning strategy to divide the sample image into n sub-image blocks, ensuring that each sub-image block includes at most one key region.

[0121] Among them, the non-uniform partitioning strategy is: perform maximum rectangle partitioning on the sample image excluding the key regions to obtain m (m is less than n) rectangular regions, and form n sub-image blocks by combining the m rectangular regions and the r key regions, where m + r = n.

[0122] Thus, each sub-image block in the sample image includes at least one connected component. The key region is determined based on the aggregation degree of all connected components. The key region includes the DNA tissue image that can reflect a certain biological structure. The distribution of the key regions of different sample images is different. Therefore, by differentially analyzing the key regions of the sample images to analyze the quality indicators of the pre- and post-image processing, the processing effect of the model on the image can be reflected more accurately and in more detail.

[0123] S305. Based on the positions of all sub-image blocks in the sample image, sub-image blocks are divided correspondingly in the class target image, which is consistent with the sample image.

[0124] In some embodiments, the influence degree value of the image quality index of the i-th sub-image block on the target evaluation index is calculated according to the following formula:

[0125]

[0126] Where, is the influence degree value of the image quality index of the i-th sub-image block on the target evaluation index, is the area value (number of pixels) of the i-th sub-image block, S is the area (total number of pixels) of the sample image, is the number of connected components in the i-th sub-image block, N is the total number of connected components in the sample image, and are preset weight factors for the importance of the area value and the number of connected components to the influence degree value, and are set according to the actual situation. .

[0127] The technical solutions in the embodiments of the present application above have at least the following technical effects or advantages:

[0128] Through the adaptive segmentation mechanism, the key information features in the fluorescence microscope image can be captured more accurately. During the training process of the image processing model, by differentially analyzing the key regions of the sample image and corresponding them to the class target image, the target evaluation index from local analysis to global can be more accurately and in more detail, reflecting the processing effect of the model on the image, improving the flexibility and accuracy of image processing, especially for images containing complex biological structures.

[0129] Example 4: In the image processing of a fluorescence microscope, due to the complexity and diversity of sample images, the global image processing model may not fully adapt to the characteristics of all local regions, resulting in local image loss or anomalies. Especially in the case of a large number of connected regions with relatively small proportions, it is difficult to avoid the possibility of these connected regions being merged when inputting a new image to be processed into the trained image processing model for global processing, resulting in the loss of local image detail content and the output of corresponding larger connected regions. There may also be a probability of image loss in abnormal regions based solely on pixel-level judgment of the front and back images.

[0130] Therefore, the embodiments of the present application are optimized to a certain extent based on the above embodiments.

[0131] In some embodiments, after step S102, the method further includes:

[0132] S103, using an adaptive segmentation mechanism to divide the image to be processed into several sub-image blocks, and assigning a unique position label to each sub-image block (which can be set according to the central pixel coordinates of the sub-image block).

[0133] S104, based on the position label of the sub-image block of the image to be processed, determine the sub-image block with the same position label in its target image, and calculate the maximum area value of the connected regions in the sub-image blocks with this position label in the image to be processed and the target image respectively, to obtain a first area value and a second area value.

[0134] S105, based on each sub-image block in the target image, calculate its image quality index and the maximum area domain difference (the difference between the second area value and the first area value), and determine whether there are abnormal sub-image blocks.

[0135] Among them, determining whether there are abnormal sub-image blocks specifically includes:

[0136] If the image quality index is less than a pre-set quality threshold and the maximum area domain difference is greater than a pre-set difference threshold, mark the position label of the corresponding sub-image block as abnormal. Among them, the quality threshold and the difference threshold are set according to expert experience and actual situations, and the difference threshold is greater than 0.

[0137] S106, if any, segment the abnormal sub-image block from the image to be processed to obtain a sub-sample, execute step S102, input it into a multi-class image processing model, output the sub-target image corresponding to the sub-sample, and according to its position label, replace the sub-image block at this position label in the target image to obtain a new target image, replacing the original target image, effectively repairing the problem of local or detail loss.

[0138] The technical solutions in the embodiments of the present application at least have the following technical effects or advantages:

[0139] By performing local differential anomaly analysis on the target image output from the image to be processed through the adaptive segmentation mechanism, the image can be processed and analyzed more precisely, improving the accuracy of image processing and the ability to capture details; by comparing the image quality index and the maximum connected domain area value of the sub-image blocks, the recognition model identifies the abnormal sub-image blocks caused by the global processing of the image to be processed, and singles out the local anomalies and inputs them into the pre-trained multi-class image processing module for targeted image processing in the local area, effectively enhancing the robustness of the model, retaining more image detail content, and making the output image more stable and reliable.

[0140] Furthermore, the embodiment of the present invention also provides an image processing device based on fluorescence microscope acquisition.

[0141] Figure 2 It is a structural block diagram of the image processing device based on fluorescence microscope acquisition in the embodiment of the present invention.

[0142] As Figure 2 shown, the image processing device based on fluorescence microscope acquisition includes: an acquisition module and an image processing module. The image processing module includes a pre-trained multi-class image processing model, and the multi-class image processing model includes: an image attention distribution module, an image distributed processing module, and an image feature fusion output module.

[0143] Specifically, the acquisition module is used to: obtain the image to be processed acquired by the fluorescence microscope, generate the acquisition environment distribution feature vector of the image to be processed, and input it into the image processing module;

[0144] The image processing module is used to: input the image to be processed and its acquisition environment distribution feature vector into the pre-trained multi-class image processing model, and output the target image corresponding to the image to be processed;

[0145] The multi-class image processing model is used to: A1. Transmit the received image to be processed and its acquisition environment distribution feature vector to the image attention distribution module to generate the distributed processing value sequence corresponding to the image to be processed, and transmit it to the image distributed processing module; A2. The image distributed processing module inputs the image to be processed into the corresponding class image processing model according to the received distributed processing value sequence, outputs several classes of target images, and transmits them to the image feature fusion output module; A3. The image feature fusion output module performs weighted fusion on the received several classes of target images according to the distributed processing value sequence to output the final target image.

[0146] It should be noted that other specific implementation contents of the embodiment of the present invention can refer to the above-mentioned image processing method based on fluorescence microscope acquisition.

[0147] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An image processing method based on fluorescence microscopy acquisition, characterized in that: include: S101, obtaining an image to be processed acquired by a fluorescence microscope, and generating an acquisition environment distribution feature vector of the image to be processed; S102, inputting the image to be processed and its acquisition environment distribution feature vector into a pre-trained multi-class image processing model, and outputting a target image corresponding to the image to be processed; Among them, the pre-trained multi-class image processing model includes: an image attention distribution module, an image distributed processing module, and an image feature fusion output module. The image distributed processing module includes several types of image processing models, each type of image processing model is assigned a unique image class label, and the image class label is determined according to the center point of the acquisition environment distribution feature vector of the training image set in the corresponding class image processing model; the image attention distribution module is specifically used for: B1. Obtain image class labels of all image processing models in the image distributed processing module; B2. Calculate the similarity values ​​of the acquisition environment distribution feature vector of the image to be processed and all image class labels respectively; B3, all the calculated similarity values ​​are sequentially combined into a distributed processing value sequence, and each similarity value in the distributed processing value sequence corresponds to a corresponding class image processing model; The multi-class image processing model is used to: A1. The received image to be processed and its acquisition environment distribution feature vector are transmitted to the image attention distribution module, a distributed processing value sequence corresponding to the image to be processed is generated, and transmitted to the image distributed processing module; A2, the image distributed processing module inputs the images to be processed into the corresponding class image processing models according to the received distributed processing value sequence, outputs several classes of target images, and transmits them to the image feature fusion output module; A3. The image feature fusion output module performs weighted fusion on the received target images of several categories according to the distributed processing value sequence, and outputs the final target image. The weight value corresponding to each category of target image corresponds to the similarity value of the target image of that category in the distributed processing value sequence.

2. The image processing method based on fluorescence microscopy acquisition according to claim 1, characterized in that: The training process of the several types of image processing models specifically includes: C1. Collect a large number of sample images collected by fluorescence microscope and generate the collection environment distribution feature vector of the sample images; C2. Input all collected acquisition environment distribution feature vectors into a preset clustering algorithm to obtain k clusters, and form a training image set with sample images corresponding to all acquisition environment distribution feature vectors in each cluster, that is, each cluster corresponds to a training image set; C3, constructing a k-type image processing model having a model structure consisting of a plurality of feature extraction layers and a plurality of convolutional transposition layers as the class image processing model to be trained; C4, based on each training image set, input the sample image therein into the corresponding class image processing model to be trained, and output the class target image; C5, based on the class target image output by each class image processing model and the sample image input, the algorithm is determined by using the preset model optimization index to generate the target evaluation index; C6. Use the target evaluation index to train the image processing model. Repeat steps C4 to C6 in a loop, continuously iterate the training, and gradually reach the target state of the target evaluation index.

3. The image processing method based on fluorescence microscopy acquisition according to claim 2, characterized in that: The preset model optimization index determination algorithm specifically includes: S201, the target image and the input sample image are divided into n sub-image blocks, and the feature vector of each sub-image block is calculated. The sub-image block feature vector of the sample image is , the sub-image block feature vector of the target image is and One to one correspondence; S202, calculating the mutual information between the sample image and the corresponding sub-image block in the class target image on the feature vector, and calculating the visual information fidelity of each sub-image block in the class target image relative to the sample image according to the following formula: ； in, For visual information fidelity, is the feature vector of the i-th sub-image block of the sample image, is the feature vector of the i-th sub-image block of the target image, for and The mutual information between for Mutual information with itself; S203, based on each sub-image block in the class target image, calculating the pixel difference between it and the corresponding sub-image block in the sample image; the pixel difference is set as: taking the variance value of the pixel difference values ​​corresponding to all pixel grids in the sub-image blocks of the class target image and the sample image; S204, determining the ratio of the visual information fidelity to the pixel difference as the covariance direction value, and obtaining the target evaluation index by combining the visual information fidelity.

4. The image processing method based on fluorescence microscopy acquisition according to claim 3, characterized in that: The target evaluation index is calculated according to the following formula: ； ； Among them, L is the target evaluation index, n is the number of sub-image blocks divided into target images, is the image quality index of the i-th sub-image block in the target image, is the visual information fidelity of the i-th sub-image block in the target image, is the covariance direction value of the i-th sub-image block in the target image, is a pre-set weight parameter used to adjust the influence of the covariance direction value on the image quality index. is the influence degree of the image quality index of the i-th sub-image block on the target evaluation index.

5. The image processing method based on fluorescence microscopy acquisition according to claim 4, characterized in that: In S201, the sub-image block division method is generated according to a preset adaptive segmentation mechanism, and the adaptive segmentation mechanism specifically includes: S301, identifying connected domains on the sample image to obtain a number of connected domains; S302, based on each connected domain in the sample image, taking the pixel coordinates of the central pixel point in the sample image as the position coordinates of the connected domain, respectively calculating the distance values ​​between any two connected domains; S303, based on the distance values ​​between the two connected domains, a preset clustering algorithm is used to obtain a plurality of clusters, each cluster including at least one connected domain, and the maximum circumscribed rectangle of each cluster is used as a key area; S304, based on all the key areas identified in the sample image, divide the sample image into a number of sub-image blocks using a non-uniform division strategy, ensuring that each sub-image block includes at most one key area; S305 , based on the positions of all sub-image blocks of the sample image, the sub-image blocks are divided in the class target image to be consistent with the sample image.

6. The image processing method based on fluorescence microscopy acquisition according to claim 4, characterized in that: The influence degree of the image quality index of the i-th sub-image block on the target evaluation index is calculated according to the following formula: ； in, is the influence of the image quality index of the i-th sub-image block on the target evaluation index, is the area value of the i-th sub-image block, S is the area of ​​the sample image, is the number of connected domains in the i-th sub-image block, N is the total number of connected domains in the sample image, and is a pre-set weight factor used to determine the importance of the area value and the number of connected domains to the impact value. .

7. The image processing method based on fluorescence microscopy acquisition according to claim 5, characterized in that: After S102, the method further includes: S103, using the adaptive segmentation mechanism to divide the image to be processed into a number of sub-image blocks, and assigning a unique position label to each sub-image block; S104, based on the position tag of the sub-image block of the image to be processed, determining the sub-image block with the same position tag in the target image, and calculating the maximum area value of the connected domain in the sub-image block with the position tag in the image to be processed and the target image, respectively, to obtain a first area value and a second area value; S105, based on each sub-image block in the target image, calculating its image quality index and maximum area domain difference, and determining whether there is an abnormal sub-image block: if the image quality index is less than a preset quality threshold, and the maximum area domain difference is greater than a preset difference threshold, marking the position label of the corresponding sub-image block as abnormal; S106, if it exists, segment the abnormal sub-image block from the image to be processed to obtain a sub-sample, execute step S102, input it into the multi-class image processing model, output the sub-target image corresponding to the sub-sample, and replace the sub-image block at the position label in the target image according to its position label, obtain a new target image, and replace the original target image.

8. An image processing device based on fluorescence microscopy acquisition, using the image processing method based on fluorescence microscopy acquisition according to any one of claims 1 to 7, characterized in that: The device comprises: an acquisition module and an image processing module, wherein the image processing module comprises multiple types of image processing models, and the multiple types of image processing models comprise: an image attention distribution module, an image distributed processing module, and an image feature fusion output module; The acquisition module is used to obtain the image to be processed collected by the fluorescence microscope, generate the collection environment distribution feature vector of the image to be processed, and input it into the image processing module; the image processing module is used to input the image to be processed and its collection environment distribution feature vector into a pre-trained multi-class image processing model, and output the target image corresponding to the image to be processed; The image distributed processing module includes several types of image processing models, each type of image processing model is assigned a unique image class label, and the image class label is determined according to the center point of the acquisition environment distribution feature vector of the training image set in the corresponding type of image processing model; the image attention distribution module is specifically used for: B1. Obtain image class labels of all image processing models in the image distributed processing module; B2. Calculate the similarity values ​​of the acquisition environment distribution feature vector of the image to be processed and all image class labels respectively; B3, all the calculated similarity values ​​are sequentially combined into a distributed processing value sequence, and each similarity value in the distributed processing value sequence corresponds to a corresponding class image processing model; The multi-class image processing model is used for: A1, transmitting the received image to be processed and its acquisition environment distribution feature vector to the image attention distribution module, generating a distributed processing value sequence corresponding to the image to be processed, and transmitting it to the image distributed processing module; A2, the image distributed processing module inputs the image to be processed into the corresponding class image processing model according to the received distributed processing value sequence, outputs several classes of target images, and transmits them to the image feature fusion output module; A3, the image feature fusion output module performs weighted fusion on the received several classes of target images according to the distributed processing value sequence, and outputs the final target image, and the weight value corresponding to each class of target image corresponds to the similarity value of the target image of that class in the distributed processing value sequence.

Images