A differential convolution-based cell positioning method, system, and storage medium

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a cell positioning method and system based on differential convolution and a storage medium. The method of the present application comprises the following steps: inputting a cell image into a ConvNeXt backbone network based on differential convolution, and extracting feature maps A at different stages; sending the feature maps A at different stages into a multi-scale hollow differential convolution module respectively, extracting gradient information, and obtaining feature maps B at different stages; inputting the feature maps B at different stages into a feature pyramid structure, performing feature fusion, and performing deconvolution on the feature map obtained after the last fusion to obtain a density map. The present application further provides a system for the above positioning method. The method and system of the present application can more effectively utilize the local gradient information in the cell image, thereby reducing the influence of the color difference of the cells on the accuracy of the positioning result, and improving the accuracy of cell positioning. The present application has good application potential in the task of cell positioning.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a cell localization method, system, and storage medium based on differential convolution. Background Technology

[0002] Cell localization is used to predict the specific location of each cell in an image, thereby determining the number of cells contained in the image. In biology and medicine, microscopic image analysis is a very important research field, and cell localization is a crucial branch within it, enabling cell counting.

[0003] Existing cell localization methods all employ traditional 2D convolution (Vanilla Convolution). Traditional convolution mainly includes two operations: (1) sampling the local receptive region R of the input feature map X, and (2) aggregating the sampled values ​​by weighting. Therefore, the input y can be expressed as:

[0004]

[0005] Where p0 represents the center position of the local receptive field R, p n w(p) represents each value in the domain. n The parameters w(p) are learnable. More specifically, for a 3×3 convolution kernel with a dilation rate of 1, the receptive field R is represented as {(-1,-1),(-1,0),...,(0,1),(1,1)}. Traditional convolution calculates the absolute value of each pixel, so theoretically, pixels with larger values ​​have more weight. This is very unfriendly to tasks such as locating and counting cell kernels with large color differences, even if the model iteratively updates the parameters w(p) in later stages. n Even adjusting the weight distribution using methods like ) cannot avoid this problem.

[0006] It is evident that existing cell localization techniques suffer from the problem that, due to variations in cell color intensity after staining, models often overlook lighter-colored cells, leading to inaccurate localization performance. There is an urgent need in this field for methods that can mitigate these color variations and accurately locate all cells. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a cell localization method, system, and storage medium based on differential convolution. The aim is to effectively utilize local gradient information in cell images through a novel model and a multi-scale dilated differential convolution module, thereby enhancing the localization performance for light-colored cells and improving the accuracy of cell localization.

[0008] A cell localization method based on differential convolution includes the following steps:

[0009] Step 1: Input the cell image into the ConvNeXt backbone network based on differential convolution and extract the feature maps A of different stages of the ConvNeXt backbone network;

[0010] Step 2: Input the feature maps A obtained in Step 1 at different stages into the multi-scale dilated differential convolution module to extract gradient information and obtain feature maps B at different stages.

[0011] Step 3: Input the feature maps B obtained in Step 2 at different stages into the feature pyramid structure for feature fusion, and perform deconvolution on the feature map obtained in the last fusion to obtain the density map.

[0012] Preferably, in step 1, the ConvNeXt backbone network based on differential convolution is a backbone network constructed by introducing differential convolution operations into the ConvNeXt network.

[0013] Preferably, in step 1, the expression for the differential convolution is:

[0014]

[0015] Where R represents the local receptive field, p0 represents the center position of the local receptive field R, and p n w(p) represents each value in the local receptive field R. n ) represents the learnable parameters, y() represents the feature map output at the current stage, x() represents the feature map input at the current stage, and θ represents the hyperparameters set in the experiment, which are set to 0.7 by default.

[0016] Preferably, in step 1, the ConvNeXt backbone network based on differential convolution includes four stages. The original size of the cell image is denoted as H×W, and the sizes of the feature maps A obtained in the four different stages are respectively... and

[0017] Preferably, in step 2, the multi-scale dilated differential convolution module includes four differential convolution sub-modules. The four differential convolution sub-modules are designed based on a 3×3 convolution kernel, with receptive field sizes of 3×3, 5×5, 7×7 and 9×9, and dilation rates of 1, 2, 3 and 4, respectively.

[0018] Preferably, step 2 specifically includes the following steps:

[0019] Step 2.1: Adjust the dimension of the feature channels by performing a 1×1 convolution on the feature map A;

[0020] Step 2.2: Input the features processed in Step 1 into differential convolutional sub-modules with different dilation rates to obtain feature maps C output by different differential convolutional sub-modules;

[0021] Step 2.3: Fill and stitch the feature map C to obtain feature map B.

[0022] Preferably, in step 3, the feature pyramid structure fuses the feature maps B from different stages in ascending order.

[0023] Preferably, in step 3, the feature map B is subjected to feature channel adjustment after being upsampled by 2 times and convolved by 1×1, and then feature fusion is performed.

[0024] The present invention also provides a cell localization system based on differential convolution, comprising:

[0025] The input module is used to input cell images;

[0026] The calculation module is used to generate a density map according to the above positioning method;

[0027] The output module is used to output density maps;

[0028] Specifically, the calculation module includes:

[0029] A ConvNeXt backbone network based on differential convolution is used to extract feature maps A at different stages from cell images;

[0030] A multi-scale dilated differential convolution module is used to extract gradient information from the feature map A to obtain feature maps B at different stages;

[0031] A feature pyramid structure is used to fuse the feature map B, and deconvolve the feature map obtained from the last fusion to obtain a density map.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon for implementing the above-described positioning method.

[0033] This invention introduces differential convolution into the ConvNeXt network and builds a cell localization model based on it. This model can effectively utilize local gradient information to enhance the model's localization performance for light-colored cells. Furthermore, to obtain richer gradient information, this invention proposes a multi-scale dilated differential convolution module. This module uses multi-scale information provided by differential convolution with different dilation rates to expand the gradient range, and then fuses features with multi-scale gradient information, enabling the model to obtain rich gradient information. The ConvNeXt backbone network based on large-kernel differential convolution and the multi-scale dilated differential convolution module constructed by this invention can alleviate the problems caused by cell color differences in existing technologies. Therefore, this invention has great application potential in cell localization tasks.

[0034] Obviously, based on the above description of the present invention, and according to common technical knowledge and conventional methods in the field, various other modifications, substitutions, or alterations can be made without departing from the basic technical concept of the present invention.

[0035] The following detailed embodiments further illustrate the above-described content of the present invention. However, this should not be construed as limiting the scope of the present invention to the following examples. All technologies implemented based on the above-described content of the present invention fall within the scope of the present invention. Attached Figure Description

[0036] Figure 1 This is a diagram of the architecture of the ConvNeXt backbone network based on differential convolution.

[0037] Figure 2 This is a schematic diagram of the differential convolution process;

[0038] Figure 3 A schematic diagram of a multi-scale dilated differential convolution module;

[0039] Figure 4 This is a schematic diagram of the feature pyramid structure fusion strategy.

[0040] Figure 5 This is a comparison of the results of cell localization performed according to the methods of Example 1 and the comparative example. Detailed Implementation

[0041] It should be noted that the algorithms for data acquisition, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structures and circuit connections not specifically described, can all be implemented using content already disclosed in the prior art.

[0042] Example 1: Cell Localization Method Based on Differential Convolution

[0043] The cell localization method in this embodiment includes the following steps:

[0044] Step 1, as follows Figure 1 As shown, cell images are input into the ConvNeXt backbone network based on differential convolution, and feature maps A of different stages of the ConvNeXt backbone network are extracted.

[0045] The detailed architecture of the ConvNeXt backbone network based on differential convolution is as follows:

[0046] 1) Differential Convolution: Traditional convolution can be represented as:

[0047]

[0048] Where R represents the local receptive field, p0 represents the center position of the local receptive field R, and p nw(p) represents each value in the domain. n ) represents the learnable parameters. Accordingly, differential convolution can be expressed as:

[0049]

[0050] That is, for each value in the local receptive field, the value at its center point is subtracted to construct the corresponding local gradient information. The entire process can be referenced. Figure 2 As shown.

[0051] Differential convolution compensates for the shortcomings of traditional convolution in representing local gradient information, enabling the model to pay more attention to cell edge information and thus notice lighter-colored cells. In cell localization tasks, considering that traditional convolution often provides stronger semantic information, this embodiment ultimately combines traditional convolution with differential convolution, such as... Figure 2 As shown, the final differential convolution is obtained as follows:

[0052]

[0053] Where R represents the local receptive field, p0 represents the center position of the local receptive field R, and p n w(p) represents each value in the local receptive field R. n ) represents the learnable parameters, y() represents the feature map output at the current stage, x() represents the feature map input at the current stage, and θ represents the hyperparameters set in the experiment, which are set to 0.7 by default.

[0054] 2) ConvNeXt Network: This network is a backbone network based on large convolutional kernels. The authors achieved performance superior to the transformer through a series of improvements using a pure convolutional architecture. The ConvNeXt network is progressively layered, mainly consisting of four stages, each corresponding to a feature map of a different scale. Assuming the original cell image size is H×W, the sizes of the four feature maps A are respectively... and The main reason for choosing this network as the backbone in this embodiment is that it uses a large number of large-size convolutional kernels, such as 7×7 convolutions. We will further explore the effect of combining large convolutional kernels with differential convolutions, which effectively enhances the ConvNeXt network's ability to perceive gradient information over a wide range.

[0055] 3) ConvNeXt backbone network based on differential convolution: To address the localization difficulties caused by differences in cell color intensity, differential convolution is introduced into the cell localization domain. Unlike traditional differential convolution, in order to obtain local gradient information of a larger receptive field, this embodiment introduces differential convolution operations into the larger convolution kernels contained in the ConvNeXt network, and constructs the backbone network accordingly.

[0056] Step 2, as follows Figure 3 As shown, the feature maps A obtained in step 1 at different stages are fed into the multi-scale dilated differential convolution module to extract gradient information and obtain feature maps B at different stages.

[0057] The Multiscale Dilated Difference Convolution (MDDC) module utilizes multiscale information provided by differential convolutions with different dilation rates to expand the gradient range. It then fuses features with multiscale gradient information, resulting in a model with rich gradient information. Specifically, the MDDC module comprises four differential convolution sub-modules based on 3×3 convolution kernels, with receptive field sizes of 3×3, 5×5, 7×7, and 9×9, and dilation rates of 1, 2, 3, and 4, respectively. For example... Figure 2 In the example, 5×5DDC represents a differential convolutional module with a receptive field size of 5×5, i.e., a kernel size of 3×3 and a dilation rate of 2.

[0058] Step 2 specifically includes the following steps:

[0059] Step 2.1: Adjust the dimension of the feature channels by performing a 1×1 convolution on the feature map A;

[0060] Step 2.2: Input the features processed in Step 1 into the four differential convolutional sub-modules with different dilation rates to obtain feature maps C output by the different differential convolutional sub-modules;

[0061] Step 2.3: In order to ensure that the feature maps output by the four branches have the same dimension, the feature map C is filled, and finally the outputs of different branches are spliced ​​together to obtain feature map B.

[0062] Step 3: Input the feature maps B obtained in Step 2 at different stages into the feature pyramid structure for feature fusion, and perform deconvolution on the feature map obtained in the last fusion to obtain the density map.

[0063] Specifically, the sizes of the four feature maps are as follows: and Integrate them one by one, from smallest to largest. For example... Figure 4 As shown, firstly... and The two feature maps B are fused together. After the feature channels of the two feature maps B are adjusted by 2x upsampling and 1×1 convolution, the features are fused together. and The same feature fusion operation is performed sequentially. Finally, the features of the last layer of the feature pyramid are deconvolved twice to obtain the final output density map.

[0064] Example 2: Cell Localization System Based on Differential Convolution

[0065] The system in this embodiment is used to implement the cell localization method of Embodiment 1, including:

[0066] The input module is used to input cell images;

[0067] The calculation module is used to generate a density map according to the above positioning method;

[0068] The output module is used to output density maps.

[0069] Specifically, the computation module includes: a ConvNeXt backbone network based on differential convolution, a multi-scale dilated differential convolution module, and a feature pyramid structure.

[0070] Specifically, the detailed architecture of the ConvNeXt backbone network based on differential convolution is as follows:

[0071] 1) Differential Convolution: Traditional convolution can be represented as:

[0072]

[0073] Where R represents the local receptive field, p0 represents the center position of the local receptive field R, and p n w(p) represents each value in the domain. n ) represents the learnable parameters. Accordingly, differential convolution can be expressed as:

[0074]

[0075] That is, for each value in the local receptive field, the value at its center point is subtracted to construct the corresponding local gradient information. The entire process can be referenced. Figure 2 As shown.

[0076] Differential convolution compensates for the shortcomings of traditional convolution in representing local gradient information, enabling the model to pay more attention to cell edge information and thus notice lighter-colored cells. In cell localization tasks, considering that traditional convolution often provides stronger semantic information, this embodiment ultimately combines traditional convolution with differential convolution, such as... Figure 2 As shown, the final differential convolution is obtained as follows:

[0077]

[0078] Where R represents the local receptive field, p0 represents the center position of the local receptive field R, and p n w(p) represents each value in the local receptive field R. n) represents the learnable parameters, y() represents the feature map output at the current stage, x() represents the feature map input at the current stage, and θ represents the hyperparameters set in the experiment, which are set to 0.7 by default.

[0079] 2) ConvNeXt Network: This network is a backbone network based on large convolutional kernels. The authors achieved performance superior to the transformer through a series of improvements using a pure convolutional architecture. The ConvNeXt network is progressively layered, mainly consisting of four stages, each corresponding to a feature map of a different scale. Assuming the original cell image size is H×W, the sizes of the four feature maps A are respectively... and The main reason for choosing this network as the backbone in this embodiment is that it uses a large number of large-size convolutional kernels, such as 7×7 convolutions. We will further explore the effect of combining large convolutional kernels with differential convolutions, which effectively enhances the ConvNeXt network's ability to perceive gradient information over a wide range.

[0080] 3) ConvNeXt backbone network based on differential convolution: To address the localization difficulties caused by differences in cell color intensity, differential convolution is introduced into the cell localization domain. Unlike traditional differential convolution, in order to obtain local gradient information of a larger receptive field, this embodiment introduces differential convolution operations into the larger convolution kernels contained in the ConvNeXt network, and constructs the backbone network accordingly.

[0081] The multi-scale dilated differential convolution module utilizes multi-scale information provided by differential convolutions with different dilation rates to expand the gradient range. Then, features with multi-scale gradient information are fused, enabling the model to obtain rich gradient information. Specifically, the multi-scale dilated differential convolution module includes a 1×1 convolution, four differential convolution sub-modules, and a concatenation module. The four differential convolution sub-modules are designed based on 3×3 convolution kernels, with receptive field sizes of 3×3, 5×5, 7×7, and 9×9, and dilation rates of 1, 2, 3, and 4, respectively. For example... Figure 2 In the diagram, 5×5DDC represents a differential convolutional module with a receptive field size of 5×5, i.e., a kernel size of 3×3 and a dilation rate of 2. The concatenation module is used to fill the feature map C, and finally, the outputs of different branches are concatenated to obtain the feature map B.

[0082] The feature pyramid structure adjusts the feature maps B at different stages by upsampling by 2x and performing 1×1 convolutions on the feature channels, then fuses them in ascending order. Finally, the features of the last layer of the pyramid are deconvolved twice to obtain the final output density map.

[0083] Example 3: Computer-readable storage medium

[0084] The computer-readable storage medium of this embodiment stores a computer program for executing the cell localization method based on differential convolution of Embodiment 1. The specific steps are as follows:

[0085] Step 1, as follows Figure 1 As shown, cell images are input into the ConvNeXt backbone network based on differential convolution, and feature maps A of different stages of the ConvNeXt backbone network are extracted.

[0086] The detailed architecture of the ConvNeXt backbone network based on differential convolution is as follows:

[0087] 1) Differential Convolution: Traditional convolution can be represented as:

[0088]

[0089] Where R represents the local receptive field, p0 represents the center position of the local receptive field R, and p n w(p) represents each value in the domain. n ) represents the learnable parameters. Accordingly, differential convolution can be expressed as:

[0090]

[0091] That is, for each value in the local receptive field, the value at its center point is subtracted to construct the corresponding local gradient information. The entire process can be referenced. Figure 2 As shown.

[0092] Differential convolution compensates for the shortcomings of traditional convolution in representing local gradient information, enabling the model to pay more attention to cell edge information and thus notice lighter-colored cells. In cell localization tasks, considering that traditional convolution often provides stronger semantic information, this embodiment ultimately combines traditional convolution with differential convolution, such as... Figure 2 As shown, the final differential convolution is obtained as follows:

[0093]

[0094] Where R represents the local receptive field, p0 represents the center position of the local receptive field R, and p n w(p) represents each value in the local receptive field R. n ) represents the learnable parameters, y() represents the feature map output at the current stage, x() represents the feature map input at the current stage, and θ represents the hyperparameters set in the experiment, which are set to 0.7 by default.

[0095] 2) ConvNeXt Network: This network is a backbone network based on large convolutional kernels. The authors achieved performance superior to the transformer through a series of improvements using a pure convolutional architecture. The ConvNeXt network is progressively layered, mainly consisting of four stages, each corresponding to a feature map of a different scale. Assuming the original cell image size is H×W, the sizes of the four feature maps A are respectively... and The main reason for choosing this network as the backbone in this embodiment is that it uses a large number of large-size convolutional kernels, such as 7×7 convolutions. We will further explore the effect of combining large convolutional kernels with differential convolutions, which effectively enhances the ConvNeXt network's ability to perceive gradient information over a wide range.

[0096] 3) ConvNeXt backbone network based on differential convolution: To address the localization difficulties caused by differences in cell color intensity, differential convolution is introduced into the cell localization domain. Unlike traditional differential convolution, in order to obtain local gradient information of a larger receptive field, this embodiment introduces differential convolution operations into the larger convolution kernels contained in the ConvNeXt network, and constructs the backbone network accordingly.

[0097] Step 2, as follows Figure 3 As shown, the feature maps A obtained in step 1 at different stages are fed into the multi-scale dilated differential convolution module to extract gradient information and obtain feature maps B at different stages.

[0098] The Multiscale Dilated Difference Convolution (MDDC) module utilizes multiscale information provided by differential convolutions with different dilation rates to expand the gradient range. It then fuses features with multiscale gradient information, resulting in a model with rich gradient information. Specifically, the MDDC module comprises four differential convolution sub-modules based on 3×3 convolution kernels, with receptive field sizes of 3×3, 5×5, 7×7, and 9×9, and dilation rates of 1, 2, 3, and 4, respectively. For example... Figure 2 In the example, 5×5DDC represents a differential convolutional module with a receptive field size of 5×5, i.e., a kernel size of 3×3 and a dilation rate of 2.

[0099] Step 2 specifically includes the following steps:

[0100] Step 2.1: Adjust the dimension of the feature channels by performing a 1×1 convolution on the feature map A;

[0101] Step 2.2: Input the features processed in Step 1 into the four differential convolutional sub-modules with different dilation rates to obtain feature maps C output by the different differential convolutional sub-modules;

[0102] Step 2.3: In order to ensure that the feature maps output by the four branches have the same dimension, the feature map C is filled, and finally the outputs of different branches are spliced ​​together to obtain feature map B.

[0103] Step 3: Input the feature maps B obtained in Step 2 at different stages into the feature pyramid structure for feature fusion, and perform deconvolution on the feature map obtained in the last fusion to obtain the density map.

[0104] Specifically, the sizes of the four feature maps are as follows: and Integrate them one by one, from smallest to largest. For example... Figure 4 As shown, firstly... and The two feature maps B are fused together. After the feature channels of the two feature maps B are adjusted by 2x upsampling and 1×1 convolution, the features are fused together. and The same feature fusion operation is performed sequentially. Finally, the features of the last layer of the feature pyramid are deconvolved twice to obtain the final output density map.

[0105] Comparative Example

[0106] To verify the effectiveness of our proposed method, we removed differential convolution and the MDCC module based on differential convolution, resulting in a basic network composed of ConvNeXt and FPN. We visualized the cell localization results, as shown below. Figure 5 As shown, the figure has three columns. The leftmost column is the original cell image, the middle column is the cell localization map, and the rightmost column is the localization visualization result. The upper half shows the localization effect of the basic model, and the lower half shows the localization effect after adding the differential convolution kernel MDDC module (i.e., using the method in Example 1). It can be seen that the introduction of differential convolution and the MDDC module effectively enhance the model's ability to locate and recognize light-colored cells, thereby improving the cell localization effect.

[0107] As can be seen from the above embodiments, the present invention provides a novel method and system for cell localization. This method and system can more effectively utilize local gradient information in cell images, thereby reducing the impact of cell color differences on the accuracy of localization results and improving cell localization accuracy. The present invention has great application potential in cell localization tasks.

Claims

1. A cell localization method based on differential convolution, characterized in that, Includes the following steps: Step 1: Input the cell image into the ConvNeXt backbone network based on differential convolution and extract the feature maps A of different stages of the ConvNeXt backbone network; the ConvNeXt backbone network based on differential convolution is the backbone network constructed by introducing differential convolution operation into the ConvNeXt network; The expression for the differential convolution is: Where R represents the local receptive field, This indicates the center location of the local receptive field R. Represents each value in the local receptive field R. Here, are learnable parameters, y() is the feature map output at the current stage, and x() is the feature map input at the current stage. Hyperparameters set for the experiment; Step 2: Input the feature maps A obtained in Step 1 at different stages into the multi-scale dilated differential convolution module to extract gradient information and obtain feature maps B at different stages. Step 3: Input the feature maps B obtained in Step 2 at different stages into the feature pyramid structure for feature fusion, and perform deconvolution on the feature map obtained in the last fusion to obtain the density map; In step 1, the ConvNeXt backbone network based on differential convolution includes four stages. The original size of the cell image is denoted as H×W, and the sizes of the feature maps A obtained in the four different stages are respectively... , , and ; Step 2 specifically includes the following steps: Step 2.1: Adjust the dimension of the feature channels by performing a 1×1 convolution on the feature map A; Step 2.2: Input the features processed in Step 1 into differential convolutional sub-modules with different dilation rates to obtain feature maps C output by different differential convolutional sub-modules; Step 2.3: Fill and stitch the feature map C to obtain feature map B.

2. The positioning method according to claim 1, characterized in that: In step 2, the multi-scale dilated differential convolution module includes four differential convolution sub-modules. The four differential convolution sub-modules are designed based on a 3×3 convolution kernel, with receptive field sizes of 3×3, 5×5, 7×7 and 9×9, and dilation rates of 1, 2, 3 and 4, respectively.

3. The positioning method according to claim 1, characterized in that: In step 3, the feature pyramid structure fuses the feature maps B from different stages in ascending order.

4. The positioning method according to claim 1, characterized in that: In step 3, the feature map B is subjected to feature channel adjustment after being upsampled by 2 times and convolved by 1×1, and then feature fusion is performed.

5. A cell localization system based on differential convolution, characterized in that, include: The input module is used to input cell images; A calculation module is configured to generate a density map according to the positioning method of any one of claims 1-4; The output module is used to output density maps; Specifically, the calculation module includes: A ConvNeXt backbone network based on differential convolution is used to extract feature maps A at different stages from cell images; A multi-scale dilated differential convolution module is used to extract gradient information from the feature map A to obtain feature maps B at different stages; A feature pyramid structure is used to fuse the feature map B, and deconvolve the feature map obtained from the last fusion to obtain a density map.

6. A computer-readable storage medium, characterized in that: It stores a computer program for implementing the positioning method according to any one of claims 1-4.

Images