Electron microscope image quality evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides an electron microscope image quality evaluation method which can be applied to the technical field of biological image processing. The method comprises the following steps: carrying out region distinguishing processing on a target electron microscope image to obtain a foreground region and a background region of the target electron microscope image, and respectively generating a mask of the foreground region and a mask of the background region of the target electron microscope image; calculating an average pixel value of the foreground area by using the mask of the foreground area, and calculating an average pixel value of the background area by using the mask of the background area; calculating the standard deviation of the foreground area by using the average pixel value of the foreground area, and calculating the standard deviation of the background area by using the average pixel value of the background area; and calculating the noise-containing contrast for evaluating the quality of the target electron microscope image by using the average pixel value and the standard deviation of the foreground region and the average pixel value and the standard deviation of the background region. The invention further provides an electron microscope image quality evaluation device and equipment and a storage medium.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, specifically to the field of biological image processing technology, and more specifically to an electron microscope image quality assessment method, device, electronic device and storage medium. Background Art

[0002] After biological samples are prepared and sliced, they can be imaged using an electron microscope to obtain biological electron microscope images with nanometer-level resolution. These images can be used to distinguish membrane structures such as cell membranes and organelle membranes in biological tissues, thereby distinguishing the microscopic structure of cells. The quality of biological electron microscope images is affected by many factors, such as sample preparation technology, electron beam dwell time, and electron beam voltage, and a quantitative image quality assessment method is needed. If it is possible to evaluate the conditions under which the best imaging quality can be obtained, a shorter electron beam dwell time, that is, a faster imaging speed, and a larger imaging area can be obtained in the same time. For large-scale biological electron microscope imaging that takes several months to image, the imaging time can be shortened and the cost can be reduced.

[0003] Existing electron microscope image quality assessment methods based on contrast cannot accurately evaluate image quality when image noise is obvious. The electron microscope image quality assessment method using contrast to noise ratio (CNR) estimates the relative level of noise and cannot be directly related to the foreground and background resolution, and ignores the difference in standard deviation between the foreground and background areas in the electron microscope image, which leads to inaccurate noise assessment. Other existing electron microscope image quality assessment methods do not distinguish between foreground and background, and are general estimates of image quality, and cannot accurately describe the foreground and background resolution in the image. Summary of the invention

[0004] In view of the above problems, the present invention provides an electron microscope image quality assessment method, device, electronic device and storage medium.

[0005] According to a first aspect of the present invention, a method for evaluating electron microscope image quality is provided, comprising:

[0006] Performing region differentiation processing on the target electron microscope image to obtain a foreground region and a background region of the target electron microscope image, and generating a mask of the foreground region and a mask of the background region respectively;

[0007] The average pixel value of the foreground area is calculated using the mask of the foreground area, and the average pixel value of the background area is calculated using the mask of the background area;

[0008] The standard deviation of the foreground area is calculated using the average pixel value of the foreground area, and the standard deviation of the background area is calculated using the average pixel value of the background area;

[0009] The noise contrast used to evaluate the quality of the target electron microscope image is calculated using the average pixel value and standard deviation of the foreground area and the average pixel value and standard deviation of the background area.

[0010] According to an embodiment of the present invention, the above-mentioned region differentiation processing is performed on the target electron microscope image to obtain the foreground region and the background region of the target electron microscope image, including:

[0011] Based on user needs, the pixel points where the target object in the target electron microscope image is located are used as the foreground area, and the pixel points where the non-target object is located are used as the background area, wherein the target object includes the membrane structure of the biological cell;

[0012] The target electron microscope image is subjected to image segmentation based on a preset image morphology processing method or a machine learning method to obtain a target electron microscope image that distinguishes a foreground area from a background area, wherein the machine learning method includes a semantic segmentation neural network and a random forest classifier.

[0013] According to an embodiment of the present invention, the above-mentioned region differentiation processing of the target electron microscope image to obtain the foreground region and the background region of the target electron microscope image also includes:

[0014] When the pixel value of the foreground area is smaller than that of the background area, the maximum pixel value of the target electron microscope image is obtained, and the pixel value of the target electron microscope image is inverted using the maximum pixel value, thereby completing the preprocessing of the target electron microscope image.

[0015] According to an embodiment of the present invention, the above-mentioned steps of respectively generating a mask of a foreground area and a mask of a background area include:

[0016] Calculate the pixel median value of the target electron microscope image, and use the pixel median value to obtain the initial mask of the foreground area and the initial mask of the background area through a threshold binarization operation;

[0017] Perform a closing operation on the initial mask of the foreground area to delete the noise points in the initial mask of the foreground area; and delete the connected areas in the initial mask of the foreground area whose connected areas are smaller than a preset value to obtain an intermediate mask of the foreground area;

[0018] Deleting a region whose thickness is greater than the theoretical thickness of the target resolution object from the intermediate mask of the foreground region to obtain a mask of the foreground region;

[0019] The initial mask of the background area is subjected to an erosion operation and a dilation operation to complete an opening operation of the mask of the background area, thereby obtaining a mask of the background area after the holes are filled.

[0020] According to an embodiment of the present invention, the above-mentioned preset value is set based on the attribute value of the target electron microscope image.

[0021] According to an embodiment of the present invention, the above-mentioned deleting the area whose thickness is greater than the theoretical thickness of the target resolution object from the intermediate mask of the foreground area to obtain the mask of the foreground area includes:

[0022] Performing an erosion operation and a dilation operation on the intermediate mask of the foreground area to complete an opening operation on the intermediate mask of the foreground area, thereby obtaining a thin film of the foreground area;

[0023] When the area value of the intermediate mask in the foreground area is 1 and the area value corresponding to the thin film in the foreground area is 0, the value of the corresponding area of ​​the intermediate mask in the foreground area is set to 1;

[0024] When the area value of the intermediate mask in the foreground area is 0 and the area value corresponding to the thin film in the foreground area is 1, the value of the corresponding area of ​​the intermediate mask in the foreground area is set to 0.

[0025] According to an embodiment of the present invention, the above-mentioned calculation of the average pixel value of the foreground area by using the mask of the foreground area includes:

[0026] Calculate the pixel value of the target electron microscope image and the pixel value of the area corresponding to the mask of the foreground area, and calculate the calculation result and the sum of the pixel values ​​of the mask of the foreground area to obtain the average pixel value of the foreground area; or

[0027] The sum of the pixel values ​​of the target electron microscope image and the total number of pixels of the target electron microscope image are calculated, and the calculation result is used as the average pixel value of the foreground area.

[0028] According to an embodiment of the present invention, calculating the standard deviation of the foreground area by using the average pixel value of the foreground area includes:

[0029] Calculate the standard deviation of the foreground area using the average pixel value of the foreground area, the pixel value of the mask of the foreground area, the sum of the pixel values ​​of the mask of the foreground area, and the pixel value of the target electron microscope image; or

[0030] The standard deviation of the foreground area is calculated using the pixel value of the target electron microscope image, the total number of pixels of the target electron microscope image, and the pixel average of the target electron microscope image.

[0031] A second aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0032] The third aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0033] The electron microscope image quality assessment method provided by the present invention eliminates interference factors that affect the electron microscope image quality assessment by distinguishing the foreground and background areas of the electron microscope image and providing a generation scheme for the foreground and background area masks, and by eliminating noise points in the foreground and background areas, and uninteresting too small or too large connected areas. Therefore, the method provided by the present invention can accurately distinguish the foreground and background, so as to achieve accurate assessment of the foreground and background resolution ability. In addition, the noise-containing contrast provided by the present invention can not only intuitively assess the difference between the foreground and background, but also take noise into account in the assessment, and can include the influence of different noise levels of the foreground and background on the foreground and background resolution ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0035] Figure 1 A diagram schematically shows an application scenario of the electron microscope image quality assessment method according to an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of pixel value intensity curves of foreground and background according to an embodiment of the present invention;

[0037] Figure 3 is a flow chart of a method for evaluating electron microscope image quality according to an embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of a process decomposition of an electron microscope image quality assessment method based on distinguishing foreground and background regions according to an embodiment of the present invention;

[0039] Figure 5 A structural block diagram of an electron microscope image quality assessment device according to an embodiment of the present invention is schematically shown;

[0040] Figure 6 The block diagram of an electronic device suitable for implementing the electron microscope image quality assessment method according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0041] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0042] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0043] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0044] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0045] Figure 1 The application scenario diagram of the electron microscope image quality assessment method according to an embodiment of the present invention is schematically shown.

[0046] like Figure 1 As shown, the application scenario 100 according to this embodiment may include the field of biological electron microscope image processing technology. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0047] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0048] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0049] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0050] It should be noted that the electron microscope image quality assessment method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the electron microscope image quality assessment device provided in the embodiment of the present invention can generally be set in the server 105. The electron microscope image quality assessment method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the electron microscope image quality assessment device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0052] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The electron microscope image quality assessment method of the disclosed embodiment is described in detail.

[0053] How to shorten the imaging time and reduce the imaging cost has always been a research hotspot in the field of biological electron microscope image processing technology. The quality assessment method of electron microscope images can evaluate under what conditions the best imaging quality can be obtained, thereby achieving a shorter electron beam residence time, that is, a faster imaging speed and a larger imaging area in the same time, which is of great significance for shortening the imaging time and reducing the imaging cost in the biological electron microscope imaging process.

[0054] In the field of biological electron microscopy image processing technology, the ability to resolve membrane structures is the core criterion for judging the quality of biological electron microscopy images. Generally, membrane structures and other structures that are expected to be resolved are called foregrounds, and the substrate or dark area opposite to them is called backgrounds.

[0055] Contrast is a common evaluation method in the field of electron microscopy to distinguish the relative difference between the foreground and the background. Its expression is shown in formula (1):

[0056] (1),

[0057] Among them, I target and I background The contrast is the average pixel value (average intensity) of the foreground and background areas respectively. When the background is 0, the contrast is 1, and when the foreground and background are the same, the contrast is 0. The larger the value, the greater the relative difference between the foreground and background. Although the contrast reflects the relative difference between the foreground and background to a certain extent, the contrast evaluation imaging effect does not take into account the influence of noise, and it is impossible to accurately evaluate the image quality when the image noise is obvious.

[0058] Figure 2 is a schematic diagram of the pixel value intensity curve of the foreground and background according to an embodiment of the present invention. Figure 2 The left figure is a schematic diagram of the pixel value intensity curve when the background and foreground have a uniform standard deviation in the background-to-noise ratio. Figure 2 The right figure is a schematic diagram of the pixel value intensity curve under normal circumstances. Generally, the foreground has a larger pixel value intensity fluctuation.

[0059] like Figure 2 As shown in Figure 2, the contrast to noise ratio (CNR: Contrast to Noise Ratio) is a common evaluation index used to quantify the difference between the signal and the background in an image relative to the noise level. Its expression is shown in formula (2):

[0060] (2),

[0061] Among them, I target and I background are the average pixel values ​​(average intensity) of the foreground and background areas, respectively, and the standard deviation is used Estimate the overall noise of the image. The contrast-to-noise ratio can be used to evaluate the image quality, but the contrast-to-noise ratio estimation ignores the difference in standard deviation between the foreground and background. Usually, the foreground area has a larger numerical fluctuation than the background, and the foreground pixel value has a larger standard deviation. Therefore, the contrast-to-noise ratio cannot accurately describe the image quality. Moreover, when the noise is the same, the contrast-to-noise ratio is target and I background When the values ​​are 1, 0 and 11, 10 respectively, they have the same value. However, the foreground and background of 1 and 0 have higher contrast, which means better foreground and background resolution. Therefore, the contrast-to-noise ratio cannot be directly correlated with the foreground and background resolution.

[0062] In addition, the existing general image quality evaluation methods do not distinguish between foreground and background, and estimate the imaging quality by statistically analyzing the image variance, the sum of squares of the image gradient amplitude, etc. However, for biological electron microscope images or other electron microscope images, there are other interference factors in addition to the foreground and background, such as bright areas of no interest caused by the aggregation of stained proteins, dust contamination on the sample, etc. Only evaluating the overall imaging quality will take all pixels into account, and it is impossible to accurately evaluate the image's ability to distinguish between the foreground and background. It is necessary to accurately distinguish between the foreground and background to eliminate interference factors. In other words, the general image quality evaluation method does not distinguish between the foreground and background, and is a general estimate of image quality, which cannot accurately describe the foreground and background resolution ability in the image.

[0063] It can be seen that the existing methods cannot fully meet the need for the ability to distinguish between foreground and background in the field of biological electron microscopy. In order to solve the technical problems existing in the prior art, the present invention provides an electron microscope image quality assessment method based on the ability to distinguish between foreground and background.

[0064] Figure 3 4 is a flow chart of a method for evaluating electron microscope image quality according to an embodiment of the present invention.

[0065] like Figure 3 As shown, the electron microscope image quality assessment method includes operations S310 to S340.

[0066] In operation S310, a region differentiation process is performed on the target electron microscope image to obtain a foreground region and a background region of the target electron microscope image, and a mask of the foreground region and a mask of the background region are generated respectively.

[0067] For the target electron microscope image, it should be ensured that the foreground area is brighter than the background area, that is, the pixel value of the foreground area is greater than the pixel value of the background area.

[0068] In operation S320, an average pixel value of the foreground area is calculated using the mask of the foreground area, and an average pixel value of the background area is calculated using the mask of the background area.

[0069] In operation S330, a standard deviation of the foreground area is calculated using an average pixel value of the foreground area, and a standard deviation of the background area is calculated using an average pixel value of the background area.

[0070] In operation S340, a noise contrast for evaluating the quality of the target electron microscope image is calculated using the average pixel value and standard deviation of the foreground area and the average pixel value and standard deviation of the background area.

[0071] The electron microscope image quality assessment method provided by the present invention eliminates interference factors that affect the electron microscope image quality assessment by distinguishing the foreground and background areas of the electron microscope image and providing a generation scheme for the foreground and background area masks, and by eliminating noise points in the foreground and background areas, and uninteresting too small or too large connected areas. Therefore, the method provided by the present invention can accurately distinguish the foreground and background, so as to achieve accurate assessment of the foreground and background resolution ability. In addition, the noise-containing contrast provided by the present invention can not only intuitively assess the difference between the foreground and background, but also take noise into account in the assessment, and can include the influence of different noise levels of the foreground and background on the foreground and background resolution ability.

[0072] According to an embodiment of the present invention, the above-mentioned region differentiation processing of the target electron microscope image to obtain the foreground area and the background area of ​​the target electron microscope image includes: based on user needs, taking the pixel points where the target object in the target electron microscope image is located as the foreground area, and taking the pixel points where the non-target object is located as the background area, wherein the target object includes the membrane structure of a biological cell; performing image segmentation on the target electron microscope image based on a preset image morphological processing method or based on a machine learning method to obtain a target electron microscope image with the foreground area and the background area distinguished, wherein the machine learning method includes a semantic segmentation neural network and a random forest classifier.

[0073] The above embodiments are illustrative, and those skilled in the art can set the foreground area and the background area according to actual needs. For example, an expert marking method can be used to finely mark the foreground and background areas; or the graphics processing solution of the present invention can be used to generate training data, and then a machine learning method (such as a U-net convolutional neural network model, a random forest classifier, etc.) can be used to perform image segmentation training to achieve foreground and background segmentation and obtain a mask.

[0074] According to an embodiment of the present invention, the above-mentioned region differentiation processing of the target electron microscope image to obtain the foreground region and the background region of the target electron microscope image also includes: when the pixel value of the foreground region is smaller than the pixel value of the background region, obtaining the maximum pixel value of the target electron microscope image, and using the maximum pixel value to perform an inversion operation on the pixel value of the target electron microscope image, thereby completing the preprocessing of the target electron microscope image.

[0075] For electron microscope images , ensure that the foreground is brighter than the background (if the brightness (i.e. pixel value) of the foreground is smaller than that of the background, for example, the membrane is darker than the cytoplasm in the transmission electron microscope image of a biological sample, then you need to first negate it, i.e. , and use the preprocessed replace ).

[0076] According to an embodiment of the present invention, the above-mentioned generation of a mask of the foreground area and a mask of the background area respectively includes: calculating the pixel median value of the target electron microscope image, and using the pixel median value to obtain an initial mask of the foreground area and an initial mask of the background area respectively through a threshold binarization operation; performing a closing operation on the initial mask of the foreground area to delete noise points in the initial mask of the foreground area; and deleting the connected areas in the initial mask of the foreground area whose connected areas are less than a preset value to obtain an intermediate mask of the foreground area; deleting an area whose thickness is greater than the theoretical thickness of the target resolution object from the intermediate mask of the foreground area to obtain a mask of the foreground area; performing an erosion operation and an expansion operation on the initial mask of the background area to complete an opening operation on the mask of the background area to obtain a mask of the background area after the holes are filled.

[0077] According to an embodiment of the present invention, the above-mentioned preset value is set based on the attribute value of the target electron microscope image.

[0078] According to an embodiment of the present invention, the above-mentioned deleting the area whose thickness is greater than the theoretical thickness of the target resolution object from the intermediate mask of the foreground area to obtain the mask of the foreground area includes: performing an etching operation and a dilation operation on the intermediate mask of the foreground area to complete the opening operation of the intermediate mask of the foreground area to obtain a thin film of the foreground area; when the area value of the intermediate mask of the foreground area is 1 and the area corresponding to the thin film of the foreground area is 0, setting the value of the corresponding area of ​​the intermediate mask of the foreground area to 1; when the area value of the intermediate mask of the foreground area is 0 and the area corresponding to the thin film of the foreground area is 1, setting the value of the corresponding area of ​​the intermediate mask of the foreground area to 0.

[0079] The following is a specific embodiment and combined with the attached Figure 4 The foreground and background area mask generation process provided by the present invention is further described in detail.

[0080] Figure 4 It is a schematic diagram of the process decomposition of the electron microscope image quality assessment method based on the distinction between the foreground and background areas according to an embodiment of the present invention.

[0081] like Figure 4 As shown in FIG. 1 , the electron microscope image quality assessment method mainly includes generating foreground and background area masks, calculating the average pixel value of the foreground and background areas, calculating the standard deviation of the foreground and background areas, and evaluating the electron microscope image quality, wherein generating foreground and background area masks includes several sub-steps such as (a) to (f).

[0082] When generating foreground and background area masks:

[0083] (a) For the selected image Find the median and get .

[0084] (b) Using threshold processing to obtain the foreground and background Mask, as shown in formula (3):

[0085] (3),

[0086] Where c0 and c1 are parameters, c0 should be greater than c1, and both are positive. Display the foreground and background masks, and adjust the parameters c0 and c1 from 1 to Preserve the foreground, such as membrane structures, in The background is included in the image, such as the cytoplasm. For example, for a biological electron microscope image with a pixel pitch of 8 nanometers, c0=1.2 and c1=1 can be used.

[0087] Optionally, the threshold processing performed in steps (a) and (b) can obtain the threshold by other methods such as manually selecting a fixed threshold, calculating the image median or mean, Otsu's method automatic threshold selection and adaptive threshold, and use it to obtain the mask in step (b).

[0088] (c) Use closing operation to delete The noise in The closing operation can be expressed by formula (4):

[0089] (4),

[0090] Among them, B0 is the structural element (a matrix with 1 in the central circle or square area and 0 in other areas, and a radius of c3. For example, B can be a 3×3 matrix with all values ​​1), and ⊕ represents the expansion operation. Represents an erosion operation. (Dilation and erosion are two basic morphological operations in image processing, which respectively expand and reduce the foreground area in the image according to the structural element).

[0091] (d) Delete The connected area with less than c4 points in the (The present invention adopts the definition of 4-adjacent connectivity.) For example, for a biological electron microscope image with a pixel spacing of 8 nanometers, c4 can be 200 to exclude connected areas of non-membrane structures (such as small bright clusters formed by certain protein aggregates after staining).

[0092] (e) Delete The area with a thickness greater than the theoretical thickness of the target resolution object. In step (e), first, Generate using opening operation , as shown in formula (5):

[0093] (5),

[0094] Among them, B1 is the structural element (a matrix with the central circular area taking 1 and other elements taking 0, and the radius being c5. For example, for a biological electron microscope image with a resolution of 8 nanometers, c5 can be 12, indicating that the membrane structure thickness is less than 12×2 pixels), and ⊕ represents the expansion operation. Represents an erosion operation.

[0095] Secondly, and The area corresponding to the value 1 is set to 0. As shown in formula (6):

[0096] (6).

[0097] (f) Open the background mask and fill the small holes in it to avoid individual noise points in the background being too bright and not being included in the background during threshold processing, as shown in formula (7):

[0098] (7),

[0099] Among them, B2 is the structural element (a matrix with 1 in the central circle or square area and 0 in other areas, and a radius of c6. For example, B2 can be a 3×3 matrix with all values ​​1), and ⊕ represents the expansion operation. Represents an erosion operation.

[0100] Alternatively, in the field of biological electron microscopy, the method of the present invention can be used to distinguish membrane structures with little thickness change. When imaging with different resolutions or downsampling the original electron microscope image, the parameters in the method can be adjusted according to the resolution change to adapt. For example, for parameters that are feasible at a certain resolution, when the resolution is changed to r times (r can be a positive real number, such as 1 / 2 or 4), the area-related parameter c4 is multiplied by r 2 , the thickness related parameter c5 is multiplied by r, and other parameters can be adapted by fine-tuning.

[0101] In addition, in the distinction between foreground and background, the subsequent calculation can be performed even if some correct foreground and background areas are excluded from the mask. Only a larger electron microscope image (for example, more than 2000×2000 pixels) or multiple electron microscope images are needed to ensure that the evaluation results are statistically stable.

[0102] According to an embodiment of the present invention, the above-mentioned calculation of the average pixel value of the foreground area using the mask of the foreground area includes: operating the pixel value of the target electron microscope image with the pixel value of the area corresponding to the mask of the foreground area, and operating the operation result with the sum of the pixel values ​​of the mask of the foreground area to obtain the average pixel value of the foreground area; or operating the sum of the pixel values ​​of the target electron microscope image with the total number of pixels of the target electron microscope image, and using the operation result as the average pixel value of the foreground area.

[0103] In general, the average pixel value of the foreground and background areas is calculated and , as shown in formula (8):

[0104] (8).

[0105] According to an embodiment of the present invention, the above-mentioned calculation of the standard deviation of the foreground area using the average pixel value of the foreground area includes: calculating the standard deviation of the foreground area using the average pixel value of the foreground area, the pixel value of the mask of the foreground area, the sum of the pixel values ​​of the mask of the foreground area and the pixel value of the target electron microscope image; or calculating the standard deviation of the foreground area using the pixel value of the target electron microscope image, the total number of pixels of the target electron microscope image and the average value of the pixels of the target electron microscope image.

[0106] In general, the standard deviation of the foreground and background areas is calculated and , as shown in formula (9):

[0107] (9).

[0108] The present invention uses the noise contrast to evaluate the quality of the electron microscope image, and calculates the noise contrast (CWN: Contrast With Noise) as shown in formula (10):

[0109] (10).

[0110] Optionally, when evaluating the quality of electron microscope images, other schemes may be used. For example, after the foreground and background masks are generated, the foreground and background contrast calculation formula (i.e., Formula 1) may be used to calculate and evaluate the image quality; after the foreground and background masks are generated, the contrast-to-noise ratio calculation formula (i.e., Formula 2) may be used to calculate and evaluate the image quality. When using Formulas (1) and (2), the calculation of the average pixel and the calculation of the standard deviation are shown in Formula (11):

[0111] (11),

[0112] Among them, I ave represents the average pixel value of the electron microscope image, represents the standard deviation, and N is the number of all pixels in the electron microscope image.

[0113] Based on the above electron microscope image quality assessment method, the present invention also provides an electron microscope image quality assessment device. Figure 5 The device is described in detail.

[0114] Figure 5The structural block diagram of the electron microscope image quality assessment device according to an embodiment of the present invention is schematically shown.

[0115] like Figure 5 As shown, the electron microscope image quality assessment device 500 includes a region classification and mask generation module 510 , an average pixel value calculation module 520 , a standard deviation calculation module 530 and a noisy contrast calculation module 540 .

[0116] The region differentiation and mask generation module 510 is used to perform region differentiation processing on the target electron microscope image, obtain the foreground region and the background region of the target electron microscope image, and generate a mask of the foreground region and a mask of the background region respectively. In one embodiment, the region differentiation and mask generation module 510 can be used to perform the operation S310 described above, which will not be repeated here.

[0117] The average pixel value calculation module 520 is used to calculate the average pixel value of the foreground area using the mask of the foreground area, and calculate the average pixel value of the background area using the mask of the background area. In one embodiment, the average pixel value calculation module 520 can be used to perform the operation S320 described above, which will not be repeated here.

[0118] The standard deviation calculation module 530 is used to calculate the standard deviation of the foreground area using the average pixel value of the foreground area, and calculate the standard deviation of the background area using the average pixel value of the background area. In one embodiment, the standard deviation calculation module 530 can be used to perform the operation S330 described above, which will not be repeated here.

[0119] The noisy contrast calculation module 540 is used to calculate the noisy contrast for evaluating the quality of the target electron microscope image using the average pixel value and standard deviation of the foreground area and the average pixel value and standard deviation of the background area. In one embodiment, the noisy contrast calculation module 540 can be used to perform the operation S340 described above, which will not be repeated here.

[0120] According to an embodiment of the present invention, any multiple modules among the region differentiation and mask generation module 510, the average pixel value calculation module 520, the standard deviation calculation module 530 and the noisy contrast calculation module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the region differentiation and mask generation module 510, the average pixel value calculation module 520, the standard deviation calculation module 530 and the noisy contrast calculation module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the region differentiation and mask generation module 510, the average pixel value calculation module 520, the standard deviation calculation module 530, and the noisy contrast calculation module 540 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0121] The above method or device provided by the present invention accurately distinguishes the foreground and background of the biological electron microscope image, and elaborates on the steps of generating the foreground and background masks. The present invention provides a corresponding processing method for eliminating noise points in the foreground and background, and uninteresting too small or too large connected areas, so as to eliminate these interference factors. In this way, the foreground and background can be accurately distinguished, so that the accurate evaluation of the foreground and background resolution ability can be achieved, which is unattainable by general global evaluation methods; in addition, the noise contrast proposed by the method or device of the present invention can not only intuitively evaluate the difference between the foreground and background, but also take noise into account in the evaluation, and can include the influence of different noise levels of the foreground and background on the foreground and background resolution ability.

[0122] Specifically, when the foreground and background noise are significant, especially when When , that is, the foreground may be smaller than the background under the influence of noise. In this case, the noise contrast is less than or equal to 0, indicating that it cannot be distinguished. For another extreme case, if the foreground and background are distinguished significantly and there is no noise, the noise contrast can reach 1. Therefore, the noise contrast usually takes a value between 0 and 1. When the value is 0, it means that the foreground and background cannot be distinguished, and when the value is 1, the foreground and background are most clearly distinguished, so that the difference between the foreground and background can be intuitively evaluated.

[0123] On the other hand, noisy contrast can be degenerated into contrast plus a correction of the noise signal ratio under certain assumptions. For example, when the current background standard deviation is the same, you can use Instead, as shown in formula (12):

[0124] (12).

[0125] The above formula (12) shows that the noisy contrast takes into account both the contrast (the former) and the proportion of noise in the signal strength (the latter). If the influence of noise is not considered and the standard deviation is set to 0, the noisy contrast is completely consistent with the contrast expression. This shows that the noisy contrast not only retains the basic form of contrast, but also adds the estimation of the foreground and background noise and its correction to the contrast. In the contrast-to-noise ratio, the standard deviation is calculated for the entire image to evaluate the noise, while in the noisy contrast proposed by the method of the present invention, the standard deviation is estimated for the foreground and background respectively. When the foreground area noise (standard deviation ) or the background area noise (standard deviation ) is significant, it may affect the distinction between the foreground and background, which is consistent with the actual situation. Therefore, the method of the present invention can more accurately estimate the influence of electron microscope image noise on imaging quality (especially the foreground and background resolution) by using different standard deviation estimates for foreground and background noise.

[0126] Figure 6 The block diagram of an electronic device suitable for implementing the electron microscope image quality assessment method according to an embodiment of the present invention is schematically shown.

[0127] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0128] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0129] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0130] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0131] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0132] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the electron microscope image quality assessment method provided by the embodiment of the present invention.

[0133] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0134] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0135] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0136] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0139] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for evaluating electron microscope image quality, characterized in that: include: Performing region differentiation processing on the target electron microscope image to obtain a foreground region and a background region of the target electron microscope image, and generating a mask of the foreground region and a mask of the background region respectively; Calculating an average pixel value of the foreground area using the mask of the foreground area, and calculating an average pixel value of the background area using the mask of the background area; Calculating the standard deviation of the foreground area using the average pixel value of the foreground area, and calculating the standard deviation of the background area using the average pixel value of the background area; The noise contrast used to evaluate the quality of the target electron microscope image is calculated by using the average pixel value and standard deviation of the foreground area and the average pixel value and standard deviation of the background area.

2. The method according to claim 1, characterized in that: The target electron microscope image is subjected to region differentiation processing to obtain the foreground region and the background region of the target electron microscope image, including: Based on user needs, the pixel points where the target object in the target electron microscope image is located are used as the foreground area, and the pixel points where the non-target object is located are used as the background area, wherein the target object includes a membrane structure of a biological cell; The target electron microscope image is subjected to image segmentation based on a preset image morphology processing method or based on a machine learning method to obtain a target electron microscope image that distinguishes a foreground area from a background area, wherein the machine learning method includes a semantic segmentation neural network and a random forest classifier.

3. The method according to claim 2, characterized in that Also includes: When the pixel value of the foreground area is smaller than the pixel value of the background area, the maximum pixel value of the target electron microscope image is obtained, and the pixel value of the target electron microscope image is inverted using the maximum pixel value, thereby completing the preprocessing of the target electron microscope image.

4. The method according to claim 1, characterized in that The mask of the foreground area and the mask of the background area are generated respectively including: Calculating the pixel median value of the target electron microscope image, and using the pixel median value to obtain an initial mask of the foreground area and an initial mask of the background area through a threshold binarization operation; Performing a closing operation on the initial mask of the foreground area to delete the noise points in the initial mask of the foreground area; and deleting the connected areas in the initial mask of the foreground area whose connected areas are smaller than a preset value to obtain an intermediate mask of the foreground area; Deleting a region whose thickness is greater than the theoretical thickness of the target resolution object from the intermediate mask of the foreground region to obtain a mask of the foreground region; The initial mask of the background area is subjected to an erosion operation and a dilation operation to complete an opening operation of the mask of the background area, thereby obtaining a mask of the background area after the holes are filled.

5. The method according to claim 4, characterized in that The preset value is set based on the attribute value of the target electron microscope image.

6. The method according to claim 4, characterized in that Deleting an area whose thickness is greater than the theoretical thickness of the target resolution object from the intermediate mask of the foreground area, and obtaining the mask of the foreground area comprises: Performing an etching operation and a dilation operation on the intermediate mask of the foreground area to thereby complete an opening operation on the intermediate mask of the foreground area to obtain a thin film of the foreground area; When the area value of the intermediate mask in the foreground area is 1 and the area value corresponding to the thin film in the foreground area is 0, setting the value of the corresponding area of ​​the intermediate mask in the foreground area to 1; When the area value of the intermediate mask in the foreground area is 0 and the area value corresponding to the thin film in the foreground area is 1, the value of the corresponding area of ​​the intermediate mask in the foreground area is set to 0.

7. The method according to claim 1, characterized in that Calculating the average pixel value of the foreground area using the mask of the foreground area includes: Calculating the pixel value of the target electron microscope image with the pixel value of the area corresponding to the mask of the foreground area, and calculating the calculation result with the sum of the pixel values ​​of the mask of the foreground area to obtain the average pixel value of the foreground area; or The sum of the pixel values ​​of the target electron microscope image and the total number of pixels of the target electron microscope image are calculated, and the calculation result is used as the average pixel value of the foreground area.

8. The method according to claim 7, characterized in that Calculating the standard deviation of the foreground area using the average pixel value of the foreground area includes: Calculating the standard deviation of the foreground area by using the average pixel value of the foreground area, the pixel value of the mask of the foreground area, the sum of the pixel values ​​of the mask of the foreground area, and the pixel value of the target electron microscope image; or The standard deviation of the foreground area is calculated using the pixel value of the target electron microscope image, the total number of pixels of the target electron microscope image, and the average value of pixels of the target electron microscope image.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images