Microscopic image shadow removing method based on halo prediction, computer equipment, storage medium and program product

Abstract

The invention discloses a halo prediction-based microscopic image shadow removal method, computer equipment, a storage medium and a program product. The objective of the invention is to solve the problem that flat-field correction needs multiple groups of flat-field correction in a zoom microscope, and to solve the problem that flat-field correction fails when exposure time is prolonged or shortened. According to the method, flat field correction does not need to be carried out on the image in advance, and the influence caused by texture introduction of the used white board in the flat field correction process is reduced. In addition, according to the collected image, the halo map is predicted in real time through the model, brightness compensation is carried out on the image according to the halo map, and the compensation effect is better.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine vision, and particularly relates to a microscopic image de-shadowing technology based on halo prediction. BACKGROUND

[0002] In microscope imaging, due to the fact that the light source and the lens cannot achieve completely parallel imaging, the middle region is usually brighter and the surrounding region (especially the corner region) is darker in the picture. The usual way to deal with this phenomenon is to correct the microscope imaging using a white board, i.e. flat field correction. The basic principle of correction is that the image taken by the white board is considered to have the same imaging brightness at the center and the edge region, and the brightness value that the edge region should be compensated for under the optical system is calculated according to the difference between the actual center and edge brightness. In actual application, the edge region is compensated for brightness to achieve the effect of making it consistent with the center region in brightness. However, in a zoom microscope, the flat field correction parameters calculated originally are invalid for each change of the physical zoom optical system, and therefore the flat field correction needs to be performed again for the new physical magnification. In addition, the flat field correction will be invalid when the image exposure time changes, especially when the exposure time is too long or too short, causing overexposure or underexposure regions in the image, and the original flat field correction parameters are invalid.

[0003] The current method is mainly flat field correction, i.e. to perform flat field correction on the imaging effect under all magnifications as much as possible, but it cannot completely remove the halo effect (the shadow under different brightness of the surrounding region is called halo) in actual shooting, nor can it dynamically adjust the compensation value difference caused by different exposure times. In addition, the white board used in flat field correction cannot be completely free of any texture, and the texture on the white board used in flat field correction will also be recorded in the flat field correction parameters, which will have a texture impact on the image shot in actual application. Therefore, the method relying solely on flat field correction still has deficiencies in the final output image effect. SUMMARY

[0004] In order to solve the technical problems mentioned in the background, the present application proposes a microscopic image de-shadowing method based on halo prediction, a computer device, a storage medium and a program product.

[0005] In order to achieve the above technical purpose, the technical scheme of the present application is as follows: A microscopic image de-shadowing method based on halo prediction, comprising the following steps: Step 1: a microscope collects a batch of images, and converts the batch of images to YCrCb space; Step 2: input the image converted to YCrCb space into a trained halo map prediction model to obtain a halo map corresponding to each image; Step 3: Calculate the mean value of all halo maps in this batch to obtain the average halo map; Step 4: Query whether there is a historical halo map with the same optical path system configuration as the current image acquisition in the historical halo database, the same optical path system configuration including the same light source setting, the same microscope magnification and the same camera exposure parameter; if there is, go to step 5, if not, jump to step 6; Step 5: Weighted fusion of the average halo map obtained in step 3 and the queried historical halo map to obtain the latest average halo map, update the latest average halo map to the corresponding optical path system configuration in the historical halo database; jump to step 7; Step 6: Directly take the average halo map obtained in step 3 as the latest average halo map, and add the latest average halo map and its corresponding optical path system configuration to the historical halo database; Step 7: Subtract each acquisition image in YCrCb space from the latest average halo map to obtain the corresponding shadow-free image of each acquisition image; Step 8: Convert all shadow-free images from YCrCb space to RGB space to obtain the final image of this batch.

[0006] Further, the halo map prediction model comprises a feature extraction module, a capsule module and a transformer decoder module, the feature extraction module extracts image features from the input image, the capsule module learns the changes of global brightness and local brightness of the image according to the image features extracted by the feature extraction module, and the transformer decoder module predicts the halo result according to the image features extracted by the feature extraction module and the brightness change features learned by the capsule module.

[0007] Further, the training data set of the halo map prediction model is obtained by combining actual acquisition and artificial production; the actual acquisition mode is to load an image before flat field correction and load an image after flat field correction for the same target, and obtain the halo map by subtracting the two images; the artificial production mode is to prepare microscope scene image data and other industrial scene image data, remove the halo area around the image, and only keep the image area near the middle of the image without halo or with a small amount of halo, simulate the intensity effect of the halo by using a Gaussian function, generate halo maps with different intensities by using different Gaussian kernels, and superimpose the halo maps and the aforementioned retained image area to generate an image with halo.

[0008] Secondly, the present application also designs a computer device, the computer device includes a processor and a memory, the memory stores a computer program, and the processor is used for executing the computer program to implement the microscopic image de-shadowing method based on halo prediction.

[0009] Thirdly, the present application also designs a computer storage medium, which stores a computer program, and the computer program implements the microscopic image de-shadowing method based on halo prediction when executed on a processor.

[0010] Finally, the present application also designs a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed by a processor to implement the microscopic image de-shadowing method based on halo prediction.

[0011] The beneficial effects brought by the above technical solutions are: The present application proposes a method for predicting the halo around the image in real time by using a prediction model, aiming at the problem that multiple sets of flat field correction are needed for flat field correction in a zoom microscope, and the problem that flat field correction is invalid when the exposure time is lengthened or shortened. The present application no longer needs to perform flat field correction on the image in advance, thereby reducing the influence introduced by the texture of the whiteboard used in the flat field correction process. In addition, according to the collected image, a halo map is predicted in real time by the model, and the image is compensated for brightness according to the halo map, so that the compensation effect is better. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a structure diagram of the halo map prediction model in the embodiment; Figure 2 is an actual application flowchart in the embodiment. DETAILED DESCRIPTION

[0013] The technical solutions of the present application will be described in detail below with reference to the drawings.

[0014] The present application designs a microscopic image de-shadowing method based on halo prediction, which uses a halo map prediction model to predict the halo of the image, and then compensates for the brightness of part of the image according to the prediction result. The present application uses deep learning technology to construct a halo map prediction model, and the model structure is as follows Figure 1As shown. The model contains three modules. The first module is the feature extraction module, which can be composed of a CNN-based structure model, or other current mainstream deep learning main module. The second module is the capsule module, which is the capsule network, which was first proposed by the father of artificial intelligence Geoffrey Hinton in 2017. Its main features are the learning of the pose change characteristics of the target, the spatial hierarchical structure, and the structure model of the relationship between global and local features. The internal structure of the capsule module is the capsule, which also includes PrimCapsule and DePrimCapsule. The role of these two sub-modules is to facilitate the conversion of the traditional CNN data form into a form convenient for capsule processing. The capsule module is applied to the scene of the present application, mainly using its learning of the global and local brightness changes to improve the ability to predict the halo features. The third module is the transformer decoder, which mainly outputs the final halo result according to the image features extracted in the front and the brightness change features learned by the capsule.

[0015] After the structure of the halo prediction model is designed, the training data set of the model needs to be prepared. The actual collection and artificial production of two methods are combined to prepare the training data set. The actual collection method is to collect images with halos and images without halos. In actual application, the image after flat field correction is taken as the image without halo. Then, for the same target, one image A is collected before loading the flat field correction, and one image B is collected after loading the flat field correction. The halo image can be represented as Y = A - B. According to this formula, in actual application, the input of the deep learning model is A, and the output is Y, that is, the halo in the original image is predicted, and the final image B = A - Y is obtained, that is, the final image is obtained by subtracting the halo from the original image. It should be noted that why the deep learning model does not directly output B, but Y? This is because the final image B contains many actual image details of the target object, and it is too complex for the deep learning model to directly predict these details, which is easy to generate many bad "false details". These are not real images. Y only contains some features of the halo in the image, and the halo image is relatively simple, only containing some brightness expressions of different regions. The method of artificially producing data is to prepare more normal image data, not only microscope images, but also other industrial scene data. These image data remove the halo area around the image and only keep the middle part of the image without halo or with a small amount of halo. The image area is simulated with a Gaussian function to simulate the intensity effect of the halo, and different brightness distribution maps are generated according to different Gaussian kernels, that is, halo images with different intensities. The generated halo image and the image without halo (or with a small amount of halo) are superimposed to generate an image with halo. In this way, a large amount of A, B, and Y data is artificially produced.

[0016] In actual training, the model is not directly trained in the RGB mode on the image, but the image is processed in the YCrCb space. YCrCb is a color space commonly used in image and video processing, which is another form of image representation. It separates the brightness information from the chrominance information. Y (luminance component): represents the brightness information of the image, reflecting the brightness of the pixel, and the value range is usually 0~255 (8-bit encoding). Cr (red chrominance component): represents the difference between red and brightness (Red component minus Y), reflecting the red deviation of the pixel. Cb (blue chrominance component): represents the difference between blue and brightness (Blue component minus Y), reflecting the blue deviation of the pixel. Halo mainly manifests as brightness information adjustment of the image, so the deep learning model is used to predict the brightness component Y. That is, the input image of the model is converted from RGB to YCrCb form, and the generated halo is a graph for the brightness component Y.

[0017] When the model is trained using the training data set described above, in actual application, first, an original picture is captured, the picture has effects such as a shadow generated by a halo, the picture is converted from RGB to YCrCb to image A (YCrCb) first, and then input into the deep learning model, a halo image Y is predicted, then image B (YCrCb) = A (YCrCb) - Y, and finally, image B is converted to the RGB space, which is the final image.

[0018] In the embodiment, the actual application process can be performed according to the flowchart as shown in Figure 2 .

[0019] S1, an image A is collected and converted to YCrCb space; S2, input into a deep learning model to predict a halo image; S3, whether the current batch of continuous shooting is completed, if not, repeat S1-S2; S4, after the current batch of shooting is completed, a plurality of images and a plurality of halo images obtained by model operation are obtained, and an average halo image is obtained by averaging all the halo images of the current batch; S5, whether there is a halo image of the same optical path system in the historical halo database; S6, if the result of S5 is “yes”, that is, the halo image calculated under the same parameters is found in the historical database, then the average halo image obtained in S4 is averaged with the halo image in the historical database to obtain the final average halo image; if the result is “no”, there is no halo image under the same optical system in the historical database, then the average halo image obtained in S4 is the final average halo image; S7, using the final average halo image to calculate B (YCrCb) = A (YCrCb) - Y for all images of the current batch, and converting the image from YCrCb space to RGB space to obtain the final image; S8, recording the final average halo image to the historical halo database, and recording the optical system parameters at this time, including light source setting, magnification, exposure time and other parameters.

[0020] In addition, the application also designs a computer device, which comprises a processor and a memory, the memory stores a computer program, and the processor is used to execute the computer program to implement the microscopic image de-shadowing method based on halo prediction.

[0021] In addition, the application also designs a computer storage medium, which stores a computer program, and the computer program is executed on a processor to implement the microscopic image de-shadowing method based on halo prediction.

[0022] Finally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the microscopic image shading method based on halo prediction.

[0023] It will be understood by those skilled in the art that this invention may relate to apparatus for performing one or more of the operations described in this application. The apparatus may be specifically designed and manufactured for the desired purpose, or may include known devices in general-purpose computers with programs stored therein that can be selectively activated or reconfigured. Such computer programs may be stored in a device (e.g., computer)-readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, including but not limited to any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), random access memory (RAM), read-only memory (ROM), electrically programmable ROM, electrically erasable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic cards, or optical cards. A readable medium includes any mechanism for storing or transmitting information in a form readable by a device (e.g., computer). For example, readable media include random access memory (RAM), read-only memory (ROM), disk storage media, optical storage media, flash memory devices, signals propagated in electrical, optical, acoustic, or other forms (e.g., carrier waves, infrared signals, digital signals), etc.

[0024] Those skilled in the art will understand that each box in these structure diagrams and / or block diagrams and / or flow diagrams, as well as combinations of boxes in these structure diagrams and / or block diagrams and / or flow diagrams, can be implemented using computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing method to generate a machine, thereby creating, through execution by the processor of the computer or other programmable data processing method, methods specified in the boxes of the structure diagrams and / or block diagrams and / or flow diagrams.

[0025] It will be understood by those skilled in the art that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that the term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0026] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0027] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for removing shadows from microscopic images based on halo prediction, characterized in that, Includes the following steps: Step 1: Acquire a batch of images using a microscope and convert the batch of images to YCrCb space; Step 2: Input the image converted to YCrCb space into the trained halo prediction model to obtain the halo map corresponding to each image; Step 3: Calculate the mean of all halo images in this batch to obtain the average halo image; Step 4: Check the historical halo database to see if there is a historical halo image with the same optical path system configuration as the one acquired in this image acquisition. The same optical path system configuration includes the same light source settings, the same microscope magnification, and the same camera exposure parameters. If it exists, proceed to step 5; otherwise, proceed to step 6. Step 5: Perform a weighted fusion of the average halo map obtained in Step 3 and the historical halo map obtained from the query to obtain the latest average halo map, and update the latest average halo map to the corresponding optical path system configuration in the historical halo database; Jump to step 7; Step 6: Directly use the average halo map obtained in Step 3 as the latest average halo map, and add the latest average halo map and its corresponding optical path system configuration to the historical halo database; Step 7: Subtract the latest average halo image from each acquired image in YCrCb space to obtain the deshaded image corresponding to each acquired image; Step 8: Convert all the deshaded images from YCrCb space to RGB space to obtain the final images for this batch.

2. The method for removing shadows from microscopic images based on halo prediction according to claim 1, characterized in that, The halo prediction model includes a feature extraction module, a capsule module, and a transformer decoder module. The feature extraction module extracts image features from the input image. The capsule module learns the changes in global and local brightness of the image based on the image features extracted by the feature extraction module. The transformer decoder module predicts the halo result based on the image features extracted by the feature extraction module and the brightness change features learned by the capsule module.

3. The method for removing shadows from microscopic images based on halo prediction according to claim 1, characterized in that, The training dataset for the halo map prediction model was obtained through a combination of actual data acquisition and manual creation. The actual data acquisition method involved acquiring an image for the same target before applying flat field correction and then acquiring another image after applying flat field correction. The halo map was obtained by subtracting the two images. The manual creation method involved preparing microscope scene image data and other industrial scene image data. For these image data, areas with halos around the edges were removed, leaving only the image areas near the center of the image that had no halo or only a small amount of halo. For these retained image areas, the intensity of the halo was simulated using a Gaussian function. Halo maps of different intensities were generated using different Gaussian kernels. These halo maps were then superimposed on the aforementioned retained image areas to generate an image with a halo.

4. A computer device, the computer device comprising a processor and a memory, the memory storing a computer program, characterized in that: The processor is used to execute the computer program to implement the microscopic image deshading method based on halo prediction as described in any one of claims 1-3.

5. A computer storage medium storing a computer program, characterized in that: When the computer program is executed on the processor, it implements the microscopic image deshading method based on halo prediction as described in any one of claims 1-3.

6. A computer program product, comprising a computer program or instructions, characterized in that: When the computer program or instructions are executed by the processor, they implement the microscopic image deshading method based on halo prediction as described in any one of claims 1-3 above.

Images