Image processing method for gene sample, image processing apparatus, and storage medium

The image of gene samples is processed through the image processing model, and the images are acquired using structured light illumination and ordinary optical microscopes, and the image processing model is optimized, which solves the hardware dependence and real-time problems in the existing technology, and realizes efficient in gene sequencing instruments Super resolution image acquisition.

WO2024227398A1PCT designated stage expired Publication Date: 2024-11-07SHENZHEN SALUS BIOMED CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/088130
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-04
Filing Date
2024-04-16
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

In the existing gene sequencing technology, hardware-based super-resolution methods are time-consuming and rely on hardware conditions, and cannot be applied in real-time gene sequencing instruments. Deep learning models cannot use hardware devices to capture real super-resolution images, resulting in the quality of output results and Poor credibility.

Method used

The image to be processed is processed through the image processing model, the label image is obtained by using a structured light illumination microscope and the training image is obtained by a normal optical microscope, and iterative training is carried out to optimize the image processing model, so as to achieve super-resolution images without high hardware conditions.

Benefits of technology

It realizes the effect of real-time acquisition of super-resolution images without relying on high hardware, improves the training speed and result quality of image processing models, and is suitable for real-time genetic detection and observation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088130_07112024_PF_FP_ABST
    Figure CN2024088130_07112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present invention are an image processing method for gene samples, an image processing apparatus, and a storage medium. The image processing method comprises: acquiring an image to be processed, said image being obtained by performing image acquisition on a first sample; and inputting said image into an image processing model for image processing so as to obtain a simulated image, wherein the simulated image is a super-resolution image, the resolution of the image to be processed is smaller than that of the simulated image, the image processing model is obtained by performing optimization processing on a first image and a second image, the first image and the second image are obtained by performing image acquisition on a second sample, the first image is a super-resolution image, and the resolution of the second image is smaller than that of the first image. The image processing method helps to acquire in real time super-resolution images of samples.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing method, image processing device and storage medium for gene samples

[0001] The present invention claims priority to Chinese patent application number 202310489113.1, filed with the Patent Office of China on May 4, 2023, entitled “Image processing method, image processing device and storage medium for genetic samples”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present invention relates to the field of gene sequencing technology, and in particular to an image processing method, an image processing device and a storage medium for gene samples. Background Art

[0003] In gene sequencing, super-resolution images are obtained through image acquisition, enabling clear observation of relevant information at the microscopic level. Hardware-based super-resolution methods can effectively reconstruct super-resolution images, but the acquisition process is time-consuming and hardware-dependent, making them unsuitable for real-time gene sequencing instruments. Technical issues

[0004] To solve the existing technical problems, the present application provides an image processing method, an image processing device and a storage medium for genetic samples. Technical Solutions

[0005] According to a first aspect of the present invention, an embodiment provides an image processing method for a gene sample, comprising:

[0006] Acquire an image to be processed, where the image to be processed is obtained by collecting an image of the first sample;

[0007] The image to be processed is input into an image processing model for image processing to obtain a simulated image, wherein the simulated image is a super-resolution image, and the resolution of the image to be processed is smaller than the resolution of the simulated image. The image processing model is obtained by optimizing a first image and a second image, and the first image and the second image are obtained by performing image acquisition on a second sample. The first image is a super-resolution image, and the resolution of the second image is smaller than the resolution of the first image.

[0008] The above-mentioned image processing method obtains a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained by collecting the sample image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring high hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0009] In some embodiments, the image processing method comprises:

[0010] Acquire the first image, where the first image is used as a label image for the image processing model;

[0011] Acquire the second image, where the second image is used as a training image for the image processing model;

[0012] Inputting the training image into the image processing model for training processing to obtain an output image;

[0013] The image processing model is optimized based on the sample difference between the label image and the output image, which is helpful to clarify the optimization direction of the image processing model.

[0014] In some embodiments, acquiring the first image includes:

[0015] The first image is acquired through a structured light illumination microscope. In this way, the image information presented by the label image can have great authenticity and clarity, allowing the image processing model to learn the information distribution of the real sample.

[0016] In some embodiments, acquiring the second image includes:

[0017] The second image is acquired by an ordinary optical microscope, or

[0018] The second image is obtained by downsampling, so that the second image can be obtained more easily.

[0019] In some embodiments, the image processing method comprises:

[0020] Performing quality enhancement processing on the first image and / or the second image can help improve image quality.

[0021] In some embodiments, optimizing the image processing model based on the degree of sample difference between the label image and the output image includes:

[0022] When the difference between the label image and the output image is greater than or equal to a preset value, the training parameters of the image processing model are optimized. The difference between the label image and the output image corresponds to the degree of sample difference between the label image and the output image. The training parameters are used to train the training images in the image processing model. In this way, a specific solution for determining the degree of sample difference between the label image and the output image can be implemented.

[0023] In some embodiments, the image processing method further comprises:

[0024] After the optimization of the training parameters is completed, the training image is input into the optimized image processing model for training to obtain a new output image;

[0025] When the difference between the label image and the new output image is greater than or equal to the preset value, optimizing the training parameters again;

[0026] When the difference between the label image and the new output image is less than the preset value, it is determined that the training of the image processing model is completed. In this way, rapid iterative optimization of the image processing model can be easily achieved.

[0027] In some embodiments, the image processing method comprises:

[0028] Determine a first loss value according to the output image and the label image, where the first loss value can be determined by a pixel mean, a variance, and a covariance of the image;

[0029] Determine a second loss value according to the output image and the label image, where the second loss value can be determined by a point color maximum of the output image and a cross entropy error between the label image and the output image;

[0030] Determining the sample difference degree according to the first loss value and the second loss value can facilitate rapid convergence of the image processing model.

[0031] An image processing method for a gene sample, comprising:

[0032] Acquire an image to be processed; wherein the image to be processed is obtained by the second image acquisition unit performing image acquisition on the gene sample to be tested;

[0033] The image to be processed is input into an image processing model for image processing to obtain a simulated image, wherein the simulated image is a super-resolution image, and the resolution of the image to be processed is smaller than the resolution of the simulated image; wherein the image processing model is obtained after training with training samples, and each training sample includes a training image collected by the second image acquisition unit for the gene sample and a label image collected by the first image acquisition unit for the gene sample; and the first image acquisition unit is a structured light illumination microscope.

[0034] In certain embodiments, the label image is obtained by reconstructing multiple images collected from genetic samples in different directions and phases by modulating sinusoidal light of different phases using the structured light illumination microscope.

[0035] In some embodiments, the image processing method further includes:

[0036] Acquire a first image; the first image is a label image collected by a structured light illumination microscope for the gene sample;

[0037] Acquire a second image; the second image is a training image collected by a common optical microscope for the genetic sample;

[0038] The label image and the training image form a training sample, and the image processing model constructed based on the deep learning network is input for iterative training. In each iteration, back propagation is performed according to the loss value between the output image of the image processing model and the label image to adjust the training parameters until the loss value between the output image and the label image is less than a preset value, thereby obtaining a trained image processing model.

[0039] In certain embodiments, the loss value includes a first loss value and a second loss value;

[0040] The first loss value is determined by pixel mean, variance and covariance of the image;

[0041] The second loss value is determined by the point color maximum of the output image, the cross entropy error between the label image and the output image.

[0042] In certain embodiments, the first loss value is determined by the following formula:

[0043] Where x represents the output image, y represents the label image, LSSIM(x,y) represents the first loss value, μ x Represents the pixel mean of the output image, μ y represents the pixel mean of the label image, σ x represents the variance of the output image, σ y represents the variance of the label image, σ xy represents the covariance of the output image and the label image, c1, c2, and c3 represent constants used to control the curvature of the formula;

[0044] The second loss value is determined by the following formula:

[0045] Among them, LPSNR represents the second loss value, MAX represents the maximum value of the point color in the output image, and MSE represents the cross entropy error between the output image and the label image.

[0046] An image processing device provided by an embodiment of the present invention includes a first image acquisition unit and an image processing unit, wherein the first image acquisition unit is configured to:

[0047] Acquire an image to be processed obtained by performing image acquisition on the first sample;

[0048] The image processing unit is used for:

[0049] The image to be processed is input into an image processing model for image processing to obtain a simulated image, wherein the simulated image is a super-resolution image, and the resolution of the image to be processed is smaller than the resolution of the simulated image. The image processing model is obtained by optimizing a first image and a second image, and the first image and the second image are obtained by performing image acquisition on a second sample. The first image is a super-resolution image, and the resolution of the second image is smaller than the resolution of the first image.

[0050] An image processing device provided by an embodiment of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the image processing method described in any one of the above embodiments are implemented.

[0051] The above-mentioned image processing device obtains a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained from the sample acquisition image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring high hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0052] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of the image processing method described in any one of the above embodiments. Beneficial effects

[0053] The above-mentioned image processing method obtains a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained by collecting the sample image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring high hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0054] The above-mentioned image processing device obtains a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained from the sample acquisition image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring high hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0055] The above-mentioned computer-readable storage medium obtains a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained from the sample acquisition image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring higher hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0056] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0058] FIG1 is a flow chart of an image processing method according to an embodiment of the present invention;

[0059] FIG2 is a block diagram of an image processing device according to an embodiment of the present invention;

[0060] FIG3 is another flow chart of an image processing method according to an embodiment of the present invention;

[0061] FIG4 is a schematic diagram of another module of the image processing device according to an embodiment of the present invention;

[0062] FIG5 is another schematic diagram of modules of the image processing device according to an embodiment of the present invention.

[0063] Reference numerals: first image acquisition unit 11 , second image acquisition unit 12 , image processing unit 13 , third image acquisition unit 14 ; memory 21 , processor 22 . Modes for Carrying Out the Invention

[0064] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0065] In related applications, gene sequencing refers to the process of identifying and analyzing genetic samples at the microscopic level through specific technical means. Gene sequencers are the primary equipment used in gene sequencing. The accuracy of current gene sequencing instruments depends largely on the microscopic scale the instrument can detect, or its resolution. The higher the instrument's resolution, the smaller and clearer the target image can be, and the more information can be obtained.

[0066] In pursuit of higher resolution, super-resolution algorithms based on hardware and traditional image processing are currently available. Hardware-based super-resolution methods, such as SIM (structured illumination microscopy), can effectively utilize photons emitted by fluorescent molecules, significantly reducing the required illumination power. However, their use is extremely time-consuming and hardware-dependent, making them unsuitable for real-time gene sequencing instruments. Super-resolution algorithms based on traditional image processing are not very practical due to their fixed parameters and high reliance on prior knowledge, which introduces noise. For other datasets based on deep learning models, scaling the original image to obtain features at different image resolutions is essentially using the scaled image to fit a super-resolution dataset for model training. This makes it impossible to capture real super-resolution images and establish a realistic super-resolution dataset using hardware equipment, resulting in poor quality and reliability of the image output from the deep learning model.

[0067] Referring to FIG1 , an image processing method for an image processing model according to an embodiment of the present invention includes:

[0068] 02: Acquire the image to be processed, where the image to be processed is obtained by collecting the image of the first sample;

[0069] 03: The image to be processed is input into the image processing model for image processing to obtain a simulated image. The simulated image is a super-resolution image. The resolution of the image to be processed is smaller than the resolution of the simulated image. The image processing model is obtained by optimizing the first image and the second image. The first image and the second image are obtained by capturing the second sample. The first image is a super-resolution image, and the resolution of the second image is smaller than the resolution of the first image.

[0070] The image processing method of the embodiment of the present invention can be implemented by the image processing device 10 of the embodiment of the present invention. Specifically, referring to FIG2 , an image processing device 10 of the embodiment of the present invention includes a first image acquisition unit 11 and an image processing unit 13. The first image acquisition unit 11 is used to: acquire an image to be processed, where the image to be processed is obtained by performing image acquisition on a first sample; the image processing unit 13 is used to: input the image to be processed into an image processing model for image processing to obtain a simulated image, where the simulated image is a super-resolution image, where the resolution of the image to be processed is less than the resolution of the simulated image, where the image processing model is obtained by optimizing the first image and the second image, where the first image and the second image are obtained by performing image acquisition on a second sample, where the first image is a super-resolution image, and where the resolution of the second image is less than the resolution of the first image.

[0071] The above-mentioned image processing method and image processing device 10 obtain a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained from the sample acquisition image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring higher hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0072] It will be appreciated that, once the image processing model is trained, the trained model is used to perform real-time image processing on the acquired image to be processed to obtain the corresponding super-resolution image. In some embodiments, the training process and the image processing process of the embodiments of the present invention are performed using the same network, without using a generator and a discriminator. This eliminates the need for structural adjustments to the image processing model and allows the output results to be automatically adapted to the current adaptation task.

[0073] In addition, in some embodiments, the image to be processed may be subjected to quality enhancement processing before being processed by the trained image processing model. The first sample and the second sample may be the same gene sample or different gene samples.

[0074] Referring to FIG3 , in some embodiments, the image processing method includes:

[0075] 011: Acquire a first image, which is used as a label image for the image processing model;

[0076] 012: Acquire a second image, where the second image is used as a training image for the image processing model;

[0077] 013: Input the training image into the image processing model for training and processing to obtain the output image;

[0078] 014: Optimize the image processing model based on the degree of sample difference between the label image and the output image.

[0079] The image processing method according to an embodiment of the present invention can be implemented by an image processing device 10 according to an embodiment of the present invention. Specifically, referring to FIG4 , the image processing device 10 includes a second image acquisition unit 12, an image processing unit 13, and a third image acquisition unit 14. The second image acquisition unit 12 is configured to acquire a first image, which serves as a label image for an image processing model; the third image acquisition unit 14 is configured to acquire a second image, which serves as a training image for the image processing model; the image processing unit 13 is configured to input the training image into the image processing model for training to obtain an output image; and optimize the image processing model based on the degree of sample difference between the label image and the output image.

[0080] This will help clarify the optimization direction of the image processing model.

[0081] Specifically, in the image processing method of the embodiment of the present invention, the gene sample can be imaged by different image acquisition methods to obtain a first image and a second image respectively, and then the second image is input as a training image of the image processing model, so that the image processing model performs training processing on the training image.

[0082] In some embodiments, the training processing of the training image by the image processing model can be achieved through multiple sub-processing processes. The multiple sub-processing may include convolution processing, activation processing, normalization processing, and upsampling processing. Among them, the convolution processing can make the convolution kernel parameters variable during the training process, and can adjust the convolution kernel parameters according to the degree of sample difference between the label image and the output image. The activation processing can increase the nonlinear expression ability of the image processing model according to the image features. The normalization processing can perform normalization operations on the image data, so that when the image processing model is optimized, it can be ensured that the parameters inside the image processing model will not be too large or too small after adjustment. The upsampling processing can change the image resolution from small to large, and the pixel values ​​sampled during the processing can be calculated by the image processing model.

[0083] Accordingly, in some embodiments, the image processing unit 13 may include a convolution module, an activation module, a normalization module, and an upsampling module. The convolution module is used to perform convolution processing. The activation module is used to perform activation processing. The normalization module is used to perform normalization processing. The upsampling module is used to perform upsampling processing. It will be understood that those skilled in the art can adjust the logical processing order between the above-mentioned multiple modules according to specific needs, and correspondingly increase or decrease the number of one or more types of modules. The specific configuration of the multiple modules in the image processing unit 13 is not limited herein.

[0084] After completing a training process on the training image, the image processing model generates an output image based on the training process results. The image processing model can use the first image as the label image and compare the output image with the label image to determine the difference between the result of processing the second image by the image processing model and the first image, thereby determining the degree of sample difference between the output image and the label image.

[0085] It is understood that the degree of sample difference between the output image and the label image can reflect the training effect of the image processing model. In some embodiments, if the degree of sample difference between the label image and the output image is relatively large, it means that the training results of the image processing model cannot clearly reflect the image information of the second sample, and it can be determined that the image processing model needs to be optimized. In some embodiments, if the degree of sample difference between the label image and the output image is relatively small, it means that the training results of the image processing model can clearly reflect the image information of the second sample.

[0086] In addition, in some embodiments, multiple first images and multiple second images can be obtained from the same genetic sample, and each first image can construct a data set with one of the second images, so that different output images can be obtained respectively by constructing multiple data sets, and then the training effect of the image processing model can be comprehensively judged through multiple different comparison results.

[0087] In one embodiment, the image processing model is a deep learning network SRCNN (Super-Resolution Convolutional Neural Network).

[0088] In some embodiments, step 011 (acquiring a first image) includes:

[0089] The first image is acquired by structured light illumination microscopy.

[0090] The image processing method of the embodiment of the present invention can be implemented by the image processing device 10 of the embodiment of the present invention. Specifically, referring to FIG4 , the second image acquisition unit 12 is used to acquire a first image through a structured light illumination microscope.

[0091] In this way, the image information presented by the labeled image can have great authenticity and clarity, allowing the image processing model to learn the information distribution of real samples.

[0092] It can be understood that the images obtained by the structured light illumination microscope have the characteristics of greater brightness and richer information. Moreover, the first image obtained by constructing the structured light illumination microscope does not require the use of zoom operations, can break through the optical diffraction limit, and will not lose the true information of the genetic sample. In some embodiments, the structured light illumination microscope can capture the high-frequency information of the genetic sample by modulating sinusoidal light of different phases, and then express the high-frequency information of the genetic sample in a low-frequency band, which is convenient for the lens to capture when collecting images. In the case of collecting multiple images, the super-resolution information of the original image can be reconstructed from different directions and different phases of the multiple images to finally form the first image.

[0093] On the basis of the above, the first image is obtained through a structured light illumination microscope for training the image processing model. The running speed of the image processing model after training can be faster on the graphics processor than directly using the structured light illumination microscope. It can also be used for real-time image acquisition and processing in genetic testing instruments, which is beneficial for the observation of living cells. Compared with traditional image processing algorithms, it has better adaptability and stability.

[0094] In some embodiments, step 012 (acquiring a second image) includes:

[0095] Acquire a second image using a conventional optical microscope, or

[0096] A second image is acquired through a downsampling process.

[0097] The image processing method of the embodiment of the present invention can be implemented by the image processing device 10 of the embodiment of the present invention. Specifically, referring to FIG4 , the third image acquisition unit 14 is used to acquire the second image through a common optical microscope or through downsampling.

[0098] In this way, the acquisition of the second image can be facilitated.

[0099] Specifically, it is understood that in practical applications, genetic samples are often observed using a conventional optical microscope. To ensure timely imaging of the observed results, these images are often acquired directly. However, these images are often low-resolution and difficult to clearly represent the true information of the genetic sample. Therefore, acquiring the second image using a conventional optical microscope can accommodate practical operational methods. Using these images as input for training the image processing model allows the training results and optimization of the image processing model to be as closely aligned with practical applications as possible.

[0100] In addition, to facilitate rapid acquisition of the second image, in some embodiments, after acquiring the first image, the first image can be downsampled to obtain the corresponding second image. In some embodiments, the first image acquisition unit 11 and the third image acquisition unit 14 can be the same, or in other words, the first image acquisition unit 11 and the third image acquisition unit 14 can include the same ordinary optical microscope, or in other words, the first image acquisition unit 11 can be used to acquire the second image, and the third image acquisition unit 14 can be used to acquire the second image.

[0101] Furthermore, in some embodiments, low-resolution images can be acquired while the structured light illumination microscope is in an unfocused state.

[0102] In certain embodiments, an image processing method comprises:

[0103] Perform quality enhancement processing on the first image and / or the second image.

[0104] The image processing method of the embodiment of the present invention can be implemented by the image processing device 10 of the embodiment of the present invention. Specifically, referring to FIG4 , the image processing unit 13 is configured to perform quality enhancement processing on the first image and / or the second image.

[0105] This can help improve image quality.

[0106] Specifically, in some embodiments, the quality enhancement process may include denoising and data enhancement, which may include rotating, flipping, and mosaic enhancement of the image, thereby increasing the variability and quantity of the image.

[0107] Additionally, in some embodiments, a quality assessment may be performed on the first image and / or the second image. In one embodiment, if the imaging information of the first image determines that the first image can clearly reflect the true information of the genetic sample, then it may be determined that quality enhancement processing is not necessary for the first image. In another embodiment, if the imaging information of the second image determines that the second image can substantially reflect the true information of the genetic sample, then it may be determined that quality enhancement processing is not necessary for the second image.

[0108] In some embodiments, step 014 (optimizing the image processing model based on the degree of sample difference between the label image and the output image) includes:

[0109] When the difference value between the label image and the output image is greater than or equal to a preset value, the training parameters of the image processing model are optimized. The difference value between the label image and the output image corresponds to the degree of sample difference between the label image and the output image. The training parameters are used to train the training images in the image processing model.

[0110] The image processing method according to the embodiment of the present invention can be implemented by the image processing device 10 according to the embodiment of the present invention. Specifically, referring to FIG4 , the image processing unit 13 is configured to optimize the training parameters of the image processing model when the difference between the label image and the output image is greater than or equal to a preset value. The difference between the label image and the output image corresponds to the degree of sample difference between the label image and the output image. The training parameters are used to train the training images in the image processing model.

[0111] This is conducive to improving the training speed and training effect of the image processing model.

[0112] Specifically, in some embodiments, the preset value may indicate a certain degree of similarity between the second sample displayed in the labeled image and the second sample displayed in the output image. Determining the relationship between the difference value and the preset value can measure the difference between the labeled image and the output image, thereby providing a better parameter optimization direction for the image processing model, thereby improving training speed and training effectiveness.

[0113] In some embodiments, the image processing method further comprises:

[0114] After the optimization of the training parameters is completed, the training image is input into the optimized image processing model for training to obtain a new output image;

[0115] When the difference between the label image and the new output image is greater than or equal to the preset value, the training parameters are optimized again;

[0116] When the difference value between the label image and the new output image is less than a preset value, it is determined that the training of the image processing model is completed.

[0117] The image processing method of the embodiment of the present invention can be implemented by the image processing device 10 of the embodiment of the present invention. Specifically, referring to FIG4 , the image processing unit 13 is configured to: upon completion of optimization processing of the training parameters, input the training image into the optimized image processing model for training processing to obtain a new output image; if the difference between the label image and the new output image is greater than or equal to a preset value, optimize the training parameters again; and if the difference between the label image and the new output image is less than a preset value, determine that the training of the image processing model is complete.

[0118] In this way, it is easy to achieve rapid iterative optimization of the image processing model.

[0119] It can be understood that when it is determined that the sample difference between the label image and the output image is large, in order to enable the image processing model to ultimately output an image result with a sufficiently small sample difference from the label image, in some embodiments, the training parameters can be adjusted by backpropagating to the training parameters of the image processing model based on the difference value. In this way, by adjusting the training parameters, the image processing model will process the training image according to the adjusted training parameters during the next training process. If it is still determined that there is a large difference value, the training parameters can be adjusted again, thereby achieving the effect of iterative optimization of the image processing model.

[0120] On the premise of determining the degree of sample difference between the label image and the new output image through the difference value, the image processing model can clearly determine whether the current processing result meets expectations. If it does not meet expectations, it can directly enter the next training process without having to judge the association and difference between the output image and the label image in other aspects; if it meets expectations, it means that the optimized image processing model can achieve the effect of converting the second image into the first image, and can retain the true information of the genetic sample in the image, so that it can be determined that the optimization of the image processing model has been completed.

[0121] In certain embodiments, an image processing method comprises:

[0122] Determine a first loss value based on the output image and the label image, where the first loss value can be determined by pixel mean, variance, and covariance of the image;

[0123] Determine a second loss value based on the output image and the label image, where the second loss value can be determined by a point color maximum of the output image and a cross entropy error between the label image and the output image;

[0124] The degree of sample difference is determined according to the first loss value and the second loss value.

[0125] The image processing method according to the embodiment of the present invention can be implemented by the image processing device 10 according to the embodiment of the present invention. Specifically, referring to FIG4 , the image processing unit 13 is configured to: determine a first loss value based on the output image and the label image, where the first loss value can be determined by the pixel mean, variance, and covariance of the image; determine a second loss value based on the output image and the label image, where the second loss value can be determined by the maximum value of the point color of the output image and the cross entropy error between the label image and the output image; and determine the degree of sample difference based on the first loss value and the second loss value.

[0126] This can facilitate the rapid convergence of the image processing model.

[0127] Specifically, in one embodiment, the first loss value can be determined by the following formula:

[0128] Where x represents the output image, y represents the label image, LSSIM(x,y) represents the first loss value, μx represents the pixel mean of the output image, μy represents the pixel mean of the label image, σx represents the variance of the output image, σy represents the variance of the label image, σxy represents the covariance of the output image and the label image, and c1, c2, and c3 represent constants used to control the curvature of the formula.

[0129] In another embodiment, the first loss value may be determined by the following formula:

[0130] Among them, LPSNR represents the second loss value, MAX represents the maximum value of the point color in the output image, and MSE represents the cross entropy error between the output image and the label image.

[0131] Based on the above embodiment, in one embodiment, the difference value can be determined by the following formula: Loss = L SSIM +λL PSNR

[0132] Wherein, Loss represents the difference value, λ represents the hyperparameter weight, and λ is used to adjust the influence of the first loss value and the second loss value on the difference value. The difference value can represent the degree of sample difference between the output image and the label image, and can reflect the degree of missing image information about the second sample in the output image relative to the image information about the second sample in the label image. In one embodiment, the larger the difference value, the greater the degree of missing image information about the second sample in the output image, and the smaller the difference value, the smaller the degree of missing image information about the second sample in the output image.

[0133] 5 , an image processing device 10 provided by an embodiment of the present invention includes a memory 21 and a processor 22. The memory 21 stores a computer program. When the processor 22 executes the computer program, the steps of the image processing method of any of the above embodiments are implemented.

[0134] For example, when the computer program is executed by the processor 22, the image processing method that can be implemented includes:

[0135] 02: Acquire the image to be processed, where the image to be processed is obtained by collecting the image of the first sample;

[0136] 03: The image to be processed is input into the image processing model for image processing to obtain a simulated image. The simulated image is a super-resolution image. The resolution of the image to be processed is smaller than the resolution of the simulated image. The image processing model is obtained by optimizing the first image and the second image. The first image and the second image are obtained by capturing the second sample. The first image is a super-resolution image, and the resolution of the second image is smaller than the resolution of the first image.

[0137] The above-mentioned image processing device 10 obtains a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained from the sample acquisition image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring high hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0138] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the steps of the image processing method of any of the above embodiments.

[0139] For example, when the computer program is executed by a processor, the image processing methods that can be implemented include:

[0140] 02: Acquire the image to be processed, which is obtained by collecting the image of the first sample;

[0141] 03: The image to be processed is input into the image processing model for image processing to obtain a simulated image. The simulated image is a super-resolution image. The resolution of the image to be processed is smaller than the resolution of the simulated image. The image processing model is obtained by optimizing the first image and the second image. The first image and the second image are obtained by capturing the second sample. The first image is a super-resolution image, and the resolution of the second image is smaller than the resolution of the first image.

[0142] The above-mentioned computer-readable storage medium obtains a simulated image by performing image processing on the image to be processed through an image processing model, so that the simulated image can simulate the super-resolution image obtained from the sample acquisition image, thereby achieving the effect of obtaining a super-resolution image of the sample without requiring higher hardware conditions, which is conducive to the real-time acquisition of the super-resolution image of the sample.

[0143] The computer-readable storage medium may be provided in the image processing apparatus 10 or in other terminals. The image processing apparatus 10 may communicate with other terminals to obtain corresponding programs.

[0144] It is understood that computer-readable storage media may include: any entity or device capable of carrying a computer program, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media. A computer program includes computer program code. The computer program code may be in source code form, object code form, an executable file, or some intermediate form. Computer-readable storage media may include: any entity or device capable of carrying a computer program code, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media.

[0145] In certain embodiments of the present invention, the image processing unit 13 may be a single-chip microcomputer chip that integrates a processor, memory, and a communication module. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0148] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, features defined as "first" or "second" may explicitly or implicitly include one or more of the described features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0149] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] Of course, the invention is not limited to the above-mentioned embodiments. Those skilled in the art may make equivalent modifications or substitutions without violating the spirit of the invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An image processing method for gene samples, characterized in that: include: Acquire an image to be processed, where the image to be processed is obtained by collecting an image of the first sample; The image to be processed is input into an image processing model for image processing to obtain a simulated image, wherein the simulated image is a super-resolution image, and the resolution of the image to be processed is smaller than the resolution of the simulated image. The image processing model is obtained after optimizing the first image and the second image, and the first image and the second image are obtained by performing image acquisition on a second sample. The first image is a super-resolution image, and the resolution of the second image is smaller than the resolution of the first image.

2. The image processing method according to claim 1, characterized in that: The image processing method comprises: Acquire the first image, where the first image is used as a label image of the image processing model; Acquire the second image, where the second image is used as a training image for the image processing model; Inputting the training image into the image processing model for training processing to obtain an output image; The image processing model is optimized according to the degree of sample difference between the label image and the output image.

3. The image processing method according to claim 2, characterized in that: Acquiring the first image includes: The first image is acquired by structured light illumination microscopy.

4. The image processing method according to claim 2, characterized in that: Acquiring the second image includes: The second image is acquired by ordinary optical microscopy, or The second image is acquired through downsampling processing.

5. The image processing method according to claim 2, characterized in that: The image processing method comprises: Perform quality enhancement processing on the first image and / or the second image.

6. The image processing method according to claim 2, characterized in that: According to the sample difference between the label image and the output image, the image processing model is optimized, including: When the difference value between the label image and the output image is greater than or equal to a preset value, the training parameters of the image processing model are optimized, the difference value between the label image and the output image corresponds to the degree of sample difference between the label image and the output image, and the training parameters are used to train the training images in the image processing model.

7. The image processing method according to claim 6, characterized in that: The image processing method further comprises: After the optimization process of the training parameters is completed, the training image is input into the optimized image processing model for training process to obtain a new output image; The difference between the label image and the new output image is greater than or equal to Under the condition of the preset value, the training parameters are optimized again; When the difference value between the label image and the new output image is less than the preset value, it is determined that the training of the image processing model is completed.

8. The image processing method according to claim 2, characterized in that: The image processing method comprises: Determine a first loss value according to the output image and the label image, where the first loss value can be determined by pixel mean, variance and covariance of the image; Determine a second loss value according to the output image and the label image, wherein the second loss value can be determined by a point color maximum value of the output image and a cross entropy error between the label image and the output image; The sample difference degree is determined according to the first loss value and the second loss value.

9. An image processing method for gene samples, characterized in that: include: Acquire an image to be processed; wherein the image to be processed is obtained by the second image acquisition unit performing image acquisition on the gene sample to be tested; The image to be processed is input into an image processing model for image processing to obtain a simulated image, wherein the simulated image is a super-resolution image, and the resolution of the image to be processed is smaller than the resolution of the simulated image; wherein the image processing model is obtained after training with training samples, and each training sample includes a training image collected by the second image acquisition unit for the gene sample and a label image collected by the first image acquisition unit for the gene sample; Wherein, the first image acquisition unit is a structured light illumination microscope.

10. The image processing method according to claim 9, characterized in that: The label image is obtained by reconstructing multiple images collected from gene samples in different directions and phases by modulating sinusoidal light of different phases under the structured light illumination microscope.

11. The image processing method according to claim 9, characterized in that: Also includes: Acquire a first image; the first image is a label image collected by a structured light illumination microscope for the gene sample; acquiring a second image; The second image is a training image collected by a common optical microscope for a gene sample; The label image and the training image form a training sample, and the image processing model constructed based on the deep learning network is input for iterative training. In each iteration, back propagation is performed according to the loss value between the output image of the image processing model and the label image to adjust the training parameters until the loss value between the output image and the label image is less than a preset value, thereby obtaining a trained image processing model.

12. The image processing method according to claim 11, characterized in that: The loss value includes a first loss value and a second loss value; The first loss value is determined by the pixel mean, variance and covariance of the image; The second loss value is determined by the point color maximum of the output image, the cross entropy error between the label image and the output image.

13. The image processing method according to claim 12, characterized in that: The first loss value is determined by the following formula: Among them, x represents the output image, y represents the label image, and LSSIM(x,y) represents the first Loss value, μx represents the pixel mean of the output image, μy represents the pixel mean of the label image, σx represents the variance of the output image, σy represents the variance of the label image, σxy represents the covariance of the output image and the label image, c1, c2, c3 represent constants used to control the curvature of the formula change; The second loss value is determined by the following formula: Among them, LPSNR represents the second loss value, MAX represents the maximum value of the point color in the output image, and MSE represents the cross entropy error between the output image and the label image.

14. An image processing device, characterized in that: The device comprises a first image acquisition unit and an image processing unit, wherein the first image acquisition unit is used for: Acquire an image to be processed, where the image to be processed is obtained by collecting an image of the first sample; The image processing unit is used for: The image to be processed is input into an image processing model for image processing to obtain a simulated image, wherein the simulated image is a super-resolution image, and the resolution of the image to be processed is smaller than the resolution of the simulated image. The image processing model is obtained after optimizing the first image and the second image, and the first image and the second image are obtained by performing image acquisition on a second sample. The first image is a super-resolution image, and the resolution of the second image is smaller than the resolution of the first image.

15. An image processing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the image processing method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 13 are implemented.

Citation Information

Patent Citations

  • Image super-resolution method and device, electronic equipment and storage medium

    CN111080527A

  • Image super-resolution and model training method and device, electronic equipment and medium

    CN111080528A

  • Crack online detection method and system based on mobile device and super-resolution reconstruction segmentation network

    CN115760720A

  • Image processing method and image processing device for gene sample and storage medium

    CN116188276A

  • Systems and methods for deep learning microscopy

    US20190333199A1