Microscopic Image Organism Recognition Method Based on Collaborative Detection

Abstract

In an embodiment of the present disclosure, a microscopic image organism recognition method based on collaborative detection is provided, belonging to the technical field of image processing. Specifically, it includes: extracting three-layer image features of different scales corresponding to the microscopic image; modifying the image feature scale to be consistent with the third-layer image feature; fusing the processed first-layer image feature, the processed second-layer image feature, and the third-layer image feature by a preset method to obtain a fused feature; inputting the fused feature into a border detection module to obtain a border detection map, obtaining a center point density map, and generating a Gaussian density map; according to the center point coordinate information included in the Gaussian density map and combining the border information included in the border detection map, comparing the center point coordinates with the border coordinates in the border detection map one by one, screening out and retaining the overlapping border information, deleting the remaining borders, optimizing the recognition result, and obtaining a border detection result. Through the solution of the present disclosure, the recognition adaptability and accuracy are improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image processing technologies, and in particular, to a microscopic image organism recognition method based on collaborative detection. Background Art

[0002] The target recognition of organisms in medical microscopic images can provide data and experience for medical tests, helping doctors make correct differential diagnoses and treatment plans based on the health conditions of patients, thereby improving the efficiency of doctors' diagnoses. For example, the recognition and counting of blood cells in blood smear microscopic images can be used to diagnose whether a patient has a blood disease; and the recognition and counting of positive and negative cells in cancer cell microscopic images can be used to determine the appropriate dosage of drugs for cancer patients, etc. Therefore, the target recognition of organisms in microscopic images plays a crucial role in the medical field.

[0003] The target recognition of organisms in microscopic images based on manual inspection consumes a large amount of time and effort, and the accuracy of organism classification highly depends on the capabilities and expertise of pathologists. At the same time, due to the cumbersome procedures, errors may also occur due to factors such as physicians' subjectivity, cognitive differences, and fatigue. Therefore, the microscopic image target recognition method based on deep learning has received wide attention. Currently, the mainstream microscopic image organism target recognition methods mainly include two categories: methods based on target bounding box detection and center point detection. The method based on bounding box detection uses a large number of rectangular bounding boxes to annotate and represent the positions of different organisms, and its recognition result is a bounding box detection map, where all the recognized target organisms are annotated by bounding boxes; the recognition method based on center point detection marks a pixel in the central area of each organism to generate a Gaussian density map for training the center point regression network, and the recognition result is a center point density map, where each point represents the center point of the recognized organism.

[0004] However, microscopic images are a type of typical high-resolution images, so there will be a large amount of background noise in the images; and the organisms in microscopic images are generally not rectangular. Therefore, for the bounding box recognition method, the bounding boxes annotating organisms not only contain target feature information but also introduce a large amount of background noise, which has great interference. This makes it easy to extract the blank areas in the background images and the features of other types of organisms, and the accuracy of the extracted features is relatively low, resulting in the recognition of other irrelevant objects. However, for the center point detection method, the point annotation only represents the organism as a point, ignoring features such as shape and size, and cannot express the spatial relationship of the organisms in the image. In addition, when a large number of organisms gather and stack, it may cause the problem of center point collision. Therefore, the current methods have limitations in the problem of microscopic image organism target recognition for medical detection.

[0005] It can be seen that there is an urgent need for a microscopic image organism recognition method based on collaborative detection with high adaptive recognition accuracy. Summary of the Invention

[0006] In view of this, embodiments of the present disclosure provide a microscopic image organism recognition method based on collaborative detection, which at least partially solves the problems of poor adaptability and accuracy in the prior art.

[0007] Embodiments of the present disclosure provide a microscopic image organism recognition method based on collaborative detection, including:

[0008] Step 1: Use an encoder to extract three-layer image features of different scales corresponding to the microscopic image;

[0009] Step 2: Respectively perform multiple one-dimensional convolutions and upsampling operations on the first-layer image features and the second-layer image features to aggregate the features within the image blocks, and modify the image feature scale to be consistent with the third-layer image features. Finally, obtain the processed image features;

[0010] Step 3: Fuse the processed first-layer image features, the processed second-layer image features, and the third-layer image features using a preset method to obtain fused features;

[0011] Step 4: Input the fused features into a bounding box detection module to obtain a bounding box detection map. Calculate the center point information of each bounding box based on the bounding box position information in the bounding box detection map to obtain a center point density map. Set geometric feature constraints using prior knowledge about the shape characteristics of organisms in the microscopic image, and generate a Gaussian density map using a Gaussian kernel and in combination with this feature constraint;

[0012] Step 5: According to the center point coordinate information included in the Gaussian density map and in combination with the bounding box information included in the bounding box detection map, compare the center point coordinates with the bounding box coordinates in the bounding box detection map one by one, screen out and retain the overlapping bounding box information, delete the remaining bounding boxes, optimize the recognition result, and obtain a bounding box detection result.

[0013] According to a specific implementation manner of the embodiments of the present disclosure, the encoder includes multiple encoding layers, and the structure of each encoding layer is, in sequence, layer normalization, multi-head self-attention, layer normalization, and a multi-layer perceptron.

[0014] According to a specific implementation manner of the embodiments of the present disclosure, the calculation formula of the encoding layer is

[0015] z′m = MSA(LN(zm - 1)) + zm - 1

[0016] zm = MLP(LN(z′m)) + z′m

[0017] Among them, z′m represents the image features after MSA and LN of the image features in the (m - 1)-th block of the encoder, MSA represents multi-head self-attention, LN represents layer normalization, MLP represents multi-layer perceptron, and zm represents the image features in the m-th block of the encoder.

[0018] According to a specific implementation manner of the embodiments of the present disclosure, the preset method is a feature splicing method.

[0019] According to a specific implementation manner of the embodiments of the present disclosure, step 4 specifically includes:

[0020] Step 4.1, set a fixed aspect ratio as a geometric feature constraint according to the prior knowledge about the geometric features of the target organism to be recognized;

[0021] Step 4.2, input the fusion feature F into the small-scale target recognition module to obtain the bounding box detection map M1, and calculate the average value l of the aspect ratio of the organism according to the bounding box information in M1 j ;

[0022] Step 4.3, calculate the center point information of each bounding box according to the bounding box position information in M1 to obtain the center point density map M2, and generate the Gaussian density map M3 by using the Gaussian kernel in combination with the geometric feature constraint.

[0023] According to a specific implementation manner of the embodiments of the present disclosure, the average value l of the aspect ratio of the organism j is calculated by the formula

[0024]

[0025] where n j is the number of all target bounding boxes recognized in the image, are the aspect ratios of the length and width of each target organism respectively.

[0026] According to a specific implementation manner of the embodiments of the present disclosure, the calculation formula of the center point density map is

[0027]

[0028]

[0029] where δ is the impulse function, x i represents the position where the center point of the organism is located, δ(x - x i ) represents the impulse response function of the organism position in the image, and N is the total number of target organisms in the image.

[0030] The microscopic image organism recognition solution based on collaborative detection in the embodiments of the present disclosure includes: Step 1, using an encoder to extract three-layer image features of different scales corresponding to the microscopic image; Step 2, respectively passing the first-layer image feature and the second-layer image feature through multiple one-dimensional convolutions and upsampling operations to aggregate the features within the image blocks, and modifying the scale of the image feature to be consistent with the third-layer image feature, and finally obtaining the processed image feature; Step 3, fusing the processed first-layer image feature, the processed second-layer image feature, and the third-layer image feature by a preset method to obtain a fused feature; Step 4, inputting the fused feature into a bounding box detection module to obtain a bounding box detection map, calculating the center point information of each bounding box based on the bounding box position information in the bounding box detection map to obtain a center point density map, and setting geometric feature constraints using the prior knowledge about the shape characteristics of organisms in the microscopic image, and generating a Gaussian density map using a Gaussian kernel and combining this feature constraint; Step 5, according to the center point coordinate information included in the Gaussian density map and the bounding box information included in the bounding box detection map, comparing the center point coordinates with the bounding box coordinates in the bounding box detection map one by one, screening out and retaining the overlapping bounding box information, deleting the remaining bounding boxes, optimizing the recognition result, and obtaining a bounding box detection result.

[0031] The beneficial effects of the embodiments of the present disclosure are as follows: Through the solution of the present disclosure, for the shape characteristics of organisms in microscopic images, first, when extracting features, the large, medium, and small scale features are unified into small scale features and fused to aggregate the features of the target organism and enhance the local expression ability of the image features; second, using the biological prior knowledge that the same organism has the same aspect ratio, geometric feature constraints are added when generating the Gaussian density map in the center point detection method to filter out incorrect recognition results and improve the recognition accuracy of the target organism; finally, the bounding box detection and center point detection methods are coordinated, and their complementarity is used to assist each other to optimize the recognition result, while retaining the rich semantic knowledge and spatial position relationship of the target organism, thereby achieving the effect of improving the recognition accuracy of the target organism. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0033] Figure 1 It is a schematic flowchart of a microscopic image organism recognition method based on collaborative detection provided by the embodiments of the present disclosure;

[0034] Figure 2 It is a schematic specific implementation flowchart of a microscopic image organism recognition method based on collaborative detection provided by the embodiments of the present disclosure. Detailed implementation manners

[0035] The following describes the embodiments of the present disclosure in detail with reference to the accompanying drawings.

[0036] The following uses specific specific examples to illustrate the implementation manners of the present disclosure. Those skilled in the art can easily understand other advantages and effects of the present disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present disclosure.

[0037] It should be noted that the following describes various aspects of the embodiments within the scope of the appended claims. It should be obvious that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is illustrative only. Based on the present disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, this device and / or practice this method can be implemented using other structures and / or functions in addition to one or more of the aspects described herein.

[0038] It should also be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present disclosure schematically. The drawings only show the components related to the present disclosure, rather than being drawn according to the number, shape and size of the components in actual implementation. The type, quantity and proportion of each component in its actual implementation can be an arbitrary change, and the component layout type may also be more complex.

[0039] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0040] Existing microscopic image organism recognition methods generally extract features from the original picture to obtain three-layer image features F∈R p×d (d represents the number of channels of the image feature), where The semantics of the image features gradually strengthen, but the location information gradually becomes coarser. Then, use FPN to perform a convolution operation with a 1x1x d convolution kernel on the image feature F3, as well as an upsampling operation, to obtain the tensor F'3. Then, perform feature fusion on the corresponding positions of all channels of this tensor and F2 to obtain the tensor F'3'. Keep reconstructing until the second convolutional layer of the forward propagation. This module obtains feature maps of three scales: large, medium, and small. Finally, input the features into the bounding box recognition module to finally obtain the bounding box detection map.

[0041] Its disadvantages are as follows: First, in high-resolution microscopic images, in addition to the organisms to be recognized, there is a large amount of background noise and other microorganisms as interference. Therefore, using rectangular boxes for feature extraction in this scheme will result in the extracted features lacking pertinence and exclusivity for special organisms, and even incorrect features may be extracted. Second, organisms in microscopic images are generally small in area and similar in shape. If the recognition methods of three scales of large, medium, and small are used simultaneously, it will greatly increase the computational cost and help limited in improving the accuracy of target recognition. Finally, the original scheme only simply extracts the features of organisms within the target box, does not strengthen the local feature expression and the correlation between target organisms, nor does it pay attention to distinguishing the features of organisms from the features of background noise. In this way, if other microbial organisms contain the same features as this organism, they are very likely to be misrecognized.

[0042] Embodiments of the present disclosure provide a method for identifying organisms in microscopic images based on collaborative detection, and the method can be applied to the process of identifying microscopic images in scenarios such as medical tests.

[0043] See Figure 1 , which is a schematic flowchart of a method for identifying organisms in microscopic images based on collaborative detection provided by embodiments of the present disclosure. As Figure 1 shown, the method mainly includes the following steps:

[0044] Step 1, use an encoder to extract three layers of image features of different scales corresponding to the microscopic image;

[0045] Optionally, the encoder includes multiple encoding layers, and the structure of each encoding layer is layer normalization, multi-head self-attention, layer normalization, and multi-layer perceptron in sequence.

[0046] Furthermore, the calculation formula of the encoding layer is

[0047] z′m = MSA(LN(zm-1)) + zm-1

[0048] zm = MLP(LN(z′m)) + z′m

[0049] Among them, z′m represents the image features after the image features in the (m - 1)-th block of the encoder pass through MSA and LN, where MSA represents multi-head self-attention, LN represents layer normalization, MLP represents multi-layer perceptron, and zm represents the image features in the m-th block of the encoder.

[0050] In specific implementation, the original microscopic image can be input into the encoder, and the encoder is used to extract image features. The encoder contains m blocks, and each block consists of layer normalization (LN), multi-head self-attention (MSA), layer normalization (LN), and multi-layer perceptron (MLP). The specific calculation process is as follows:

[0051] z′ m = MSA(LN(z m-1 )) + z m-1

[0052] z m = MLP(LN(z′ m )) + z′ m

[0053] The encoder obtains three layers of image features F ∈ R p×d (d represents the number of channels of the image features), where

[0054] Step 2: Respectively perform multiple one-dimensional convolution and upsampling operations on the first-layer image features and the second-layer image features to aggregate the features within the image blocks, and modify the image feature scale to be consistent with the third-layer image features. Finally, the processed image features are obtained;

[0055] In specific implementation, the image features F i (i = 1, 2) can be respectively subjected to n i (n1 = 5, n2 = 3) one-dimensional convolution and upsampling operations to aggregate the features within the image blocks, enhance the local expression ability of the image features, where the convolution kernel is 1x1xd (d is the number of feature channels). And modify the image feature scale to obtain the processed image features F'1 ∈ R 8×d , F'2 ∈ R 8×d . The large-scale and medium-scale image features are unified into small-scale.

[0056] Step 3: Fuse the processed first-layer image features, the processed second-layer image features, and the third-layer image features by using a preset method to obtain fused features;

[0057] Optionally, the preset method is a feature splicing method.

[0058] During specific implementation, the image features F'1, F'2 and F3 obtained after processing are subjected to feature fusion, and the preset method is the feature splicing (concat) method. The fused feature F ∈ R 8×d' , where d' is the number of channels after fusion. It should be noted that other methods can be selected for the fusion method during specific use, which will not be elaborated here one by one.

[0059] Step 4: Input the fused feature into the border detection module to obtain a border detection map, calculate the center point information of each border based on the border position information in the border detection map to obtain a center point density map, and set geometric feature constraints using the prior knowledge about the shape characteristics of organisms in microscopic images, and generate a Gaussian density map using a Gaussian kernel and in combination with this feature constraint;

[0060] Based on the above embodiments, step 4 specifically includes:

[0061] Step 4.1: Set a fixed aspect ratio as the geometric feature constraint according to the prior knowledge about the geometric features of the target organism to be recognized;

[0062] Step 4.2: Input the fused feature F into the small-scale target recognition module to obtain a border detection map M1, and calculate the average value l of the aspect ratios of organisms based on the information of each border in M1 j ;

[0063] Step 4.3: Calculate the center point information of each border based on the border position information in M1 to obtain a center point density map M2, and generate a Gaussian density map M3 using a Gaussian kernel in combination with the geometric feature constraint.

[0064] Furthermore, the average value l of the aspect ratios of organisms j is calculated by the formula

[0065]

[0066] where n j is the number of all target borders recognized in the image, are the aspect ratios of the length and width of each target organism respectively.

[0067] Furthermore, the calculation formula of the center point density map is

[0068]

[0069]

[0070] where δ is the impulse function, x i represents the position where the center point of the organism is located, δ(x - x i)The impulse response function indicating the position of the organism in the image, where N is the total number of target organisms in the image.

[0071] In specific implementation, a fixed aspect ratio can be set according to the prior knowledge of the geometric features of the target organism to be recognized. Then, the image feature F is input into the small-scale target recognition module to obtain the border detection map M1, and the average value l of the aspect ratio of the organism is calculated according to the border information in M1. j , and the generation process is as follows:

[0072]

[0073] where n j is the total number of target borders recognized in the image, are the length and width ratios of each target organism respectively;

[0074] According to the center point information of each border calculated from the border position information in M1, the center point density map M2 is obtained, and the Gaussian density map M3 is generated using the Gaussian kernel. The generation process is as follows:

[0075]

[0076]

[0077] where δ is the impulse function, and x i represents the position where the center point of the organism is located, and δ(x - x i ) represents the impulse response function of the position of the organism in the image, and N is the total number of target organisms in the image.

[0078] The center point density map can be obtained through the above formula.

[0079] Step 5: According to the center point coordinate information included in the Gaussian density map and combined with the border information included in the border detection map, the center point coordinates are compared with the border coordinates in the border detection map one by one, the overlapping border information is screened out and retained, and the remaining borders are deleted to optimize the recognition result and obtain the border detection result.

[0080] In specific implementation, according to the center point coordinate information included in M3 and combined with the border information included in M1, the center point coordinates are compared with the border coordinates in it one by one, the overlapping border information is screened out and retained, and the remaining borders are deleted to optimize the recognition result and obtain the final border detection map as the border detection result.

[0081] The microscopic image organism recognition method based on collaborative detection provided in this embodiment designs a microscopic image organism recognition method based on target bounding box - center point collaborative detection by utilizing the complementarity of the bounding box and center point image detection methods. It combines the advantages of the target bounding box detection and center point detection methods, effectively solves the center point collision problem, reflects the spatial relationship of organisms, and retains rich semantic knowledge of the target, thereby improving the recognition accuracy of the target organism.

[0082] According to the characteristics of high resolution and small target organisms in microscopic images, the features of multiple scales extracted are convolved, upsampled, and finally fused into small-scale features for target recognition, so as to accurately extract organism features, enhance the local expression ability of the features of only the organism part in the microscopic image, and reduce the negative impact of background noise. In the part of target recognition using the center point detection method, by using the prior knowledge that organisms of the same type have a relatively fixed aspect ratio, the fixed aspect ratio set by experts is combined with the average aspect ratio of each target organism in the image, and then incorporated into the generation process of the Gaussian density map, ultimately achieving the purpose of optimizing the recognition result by using the geometric features of organisms.

[0083] The units involved in the embodiments described in this disclosure can be implemented in software or in hardware.

[0084] It should be understood that each part of this disclosure can be implemented by hardware, software, firmware, or a combination thereof.

[0085] As mentioned above, the above are only specific embodiments of this disclosure, but the protection scope of this disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in this disclosure should be covered by the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be subject to the protection scope of the claims.

Claims

1. A microscopic image organism recognition method based on collaborative detection, characterized in that include: Step 1, using the encoder to extract three-layer image features of different scales corresponding to the microscopic image; Step 2: The first layer image features and the second layer image features are subjected to multiple one-dimensional convolution and upsampling operations to aggregate the features within the image block, and the image feature scale is modified to be consistent with the third layer image features, and finally the processed image features are obtained; Step 3, fusing the processed first layer image features, the processed second layer image features and the third layer image features using a preset method to obtain fused features; Step 4, input the fusion features into the border detection module to obtain a border detection map, calculate the center point information of each border according to the border position information in the border detection map to obtain a center point density map, and use the prior knowledge about the appearance characteristics of the organism in the microscopic image to set the geometric feature constraints, and use the Gaussian kernel and combine the feature constraints to generate a Gaussian density map; Step 5: Based on the center point coordinate information contained in the Gaussian density map and the border information contained in the border detection map, the center point coordinates are compared with the border coordinates in the border detection map one by one, the overlapping border information is filtered out and retained, the remaining borders are deleted, the recognition results are optimized, and the border detection results are obtained.

2. The method according to claim 1, wherein ,The encoder includes multiple encoding layers, and the structure of each encoding layer is layer normalization, multi-head self-attention, layer normalization and multi-layer perceptron.

3. The method according to claim 2, wherein , the calculation formula of the coding layer is z′m=MSA(LN(zm-1))+zm-1 zm=MLP(LN(z′m))+z′m Among them, z′m represents the image features in the m-1th block encoder after MSA and LN, MSA represents multi-head self-attention, LN represents layer normalization, MLP represents multi-layer perceptron, and zm represents the image features in the mth block encoder.

4. The method according to claim 3, wherein ,The preset method is a feature stitching method.

5. The method according to claim 4, characterized in that , the step 4 specifically includes: Step 4.1, set a fixed aspect ratio as the geometric feature constraint according to the prior knowledge of the geometric features of the target organism to be recognized ; Step 4.2, input the fused feature F into the small-scale object recognition module to obtain the bounding box detection map M1, and calculate the average value l of the aspect ratio of the organism according to the bounding box information in M1 j ; Step 4.3, calculate the center point information of each border according to the border position information in M1 to obtain the center point density map M2, and use the Gaussian kernel combined with geometric feature constraints to generate the Gaussian density map M3.

6. The method according to claim 5, characterized in that , the average value l of the aspect ratio of the organism j The calculation formula is where n j is the number of all target bounding boxes recognized in the image, are the aspect ratios of the length and width of each target organism respectively.

7. The method according to claim 6, characterized in that , the calculation formula of the center point density map is: where δ is the impulse function, x i represents the position of the center point of the organism, and δ(x - x i ) represents the impulse response function of the organism position in the image, and N is the total number of target organisms in the image.

Images