Lightweight cell localization method and system
By using a lightweight cell localization model, differential convolution and attention modules are used to enhance gradient information. Combined with Ghost Bottleneck and grouped convolution, the problems of poor localization performance and slow inference speed in dense regions are solved, achieving efficient cell localization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WEST CHINA PRECISION MEDICINE IND TECH INST
- Filing Date
- 2022-11-25
- Publication Date
- 2026-05-01
AI Technical Summary
Existing cell localization methods have poor localization performance in dense areas and slow model inference speed, making it difficult to meet practical needs.
A lightweight cell localization model is adopted, which combines differential convolution and attention modules. Differential convolution enhances gradient information and attention modules are introduced for adaptive optimization. Ghost Bottleneck and grouped convolution are combined to reduce computational cost.
It improves the localization accuracy of cells in dense areas, reduces computational costs, enables the model to be applied to low-computing-power scenarios, and enhances the model's localization performance and inference speed.
Smart Images
Figure CN115810046B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cell localization technology, specifically to a lightweight cell localization method and system. Background Technology
[0002] Cell localization, or predicting the specific location of each cell nucleus in an image, is a crucial research area in biology and medicine, with cell localization being a significant branch. With the development of artificial intelligence, especially the widespread adoption of deep learning, directly predicting cell location using deep learning is gradually becoming a new auxiliary diagnostic and treatment method.
[0003] In related technologies, existing cell localization methods mainly face two major challenges: 1. Because cells are often unevenly distributed, this contradicts the strategy of existing models that treat all localities in an image equally; more specifically, existing models do not pay enough attention to densely populated areas, resulting in poor cell localization in these areas. 2. The parameter size and inference speed of existing cell localization models are insufficient for practical applications; the slow inference speed limits the further application of the models and leads to a decline in the user experience. Summary of the Invention
[0004] To at least partially overcome the problems of poor cell localization in dense areas and slow model inference speed in related technologies, this application provides a lightweight cell localization method and system.
[0005] According to a first aspect of the embodiments of this application, a lightweight cell localization method is provided, comprising:
[0006] The image to be processed is input into the cell localization model; wherein the cell localization model is a model obtained after lightweight processing, and differential convolution and attention modules are introduced into the cell localization model;
[0007] At the beginning of the initial stage of the cell localization model, the gradient information of the image is enhanced by differential convolution to obtain a feature map containing gradient information.
[0008] In subsequent stages of the cell localization model, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized through an attention module to obtain an optimized feature map.
[0009] Cell localization information is obtained based on the optimized feature map.
[0010] Furthermore, the method for building the cell localization model is as follows: a U-Net network is used, differential convolution is introduced at the front end of the feature encoding part of the U-Net network, and an attention module is introduced at the end of each stage of the feature encoding part.
[0011] Furthermore, differential convolution is introduced at the very front of the feature encoding part of the U-Net network, including the following steps: replacing the two convolutional layers at the very front of the U-Net network with differential convolution.
[0012] Furthermore, the attention module is used to: optimize the feature map, send the optimized feature map to the next stage, and laterally jump to the corresponding decoding stage.
[0013] Furthermore, the lightweighting process of the cell localization model includes the following steps:
[0014] The number of channels performing convolution operations at each stage in the cell localization model is truncated.
[0015] Grouped convolution is introduced into the convolution operation of the cell localization model;
[0016] The Ghost Bottleneck module is introduced into the cell localization model.
[0017] Furthermore, the number of channels performing convolution operations at each stage in the cell localization model is pruned, including:
[0018] The number of channels in each stage of the cell localization model was cropped to [3,32,64,128,256,512,256,128,64,32,1].
[0019] Furthermore, introducing grouped convolution into the convolution operation of the U-Net network includes the following steps:
[0020] The feature map is divided into multiple groups according to channels, and then convolution is performed on each group.
[0021] According to a second aspect of the embodiments of this application, a lightweight cell localization system is provided, comprising: an input module and a cell localization model; the cell localization model is a model obtained after lightweight processing, and differential convolution and attention modules are introduced into the cell localization model;
[0022] The input module is used to input the image to be processed into the cell localization model;
[0023] The cell localization model is used in the initial stage to enhance the gradient information of the image through differential convolution to obtain a feature map containing gradient information; in subsequent stages, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized through an attention module to obtain an optimized feature map; cell localization information is obtained based on the optimized feature map.
[0024] According to a third aspect of the embodiments of this application, a computer device is provided, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program in the memory to implement the operation steps of the method as described in any of the above embodiments.
[0025] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the operation steps of the method as described in any of the above embodiments.
[0026] The technical solutions provided by the embodiments of this application have the following beneficial effects:
[0027] The proposed solution introduces an attention module into the cell localization model, enabling it to notice densely packed cells in the scene. This significantly enhances the model's localization performance while increasing computational cost slightly. Differential convolution is also introduced into the model to effectively reduce the absolute value difference between pixels, allowing the model to focus more on gradient information between images and thus improving the model's accuracy in recognizing lighter-colored cells. Furthermore, the cell localization model is lightweighted to reduce its computational cost, making it applicable to more low-computing scenarios.
[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0030] Figure 1 This is a flowchart illustrating a lightweight cell localization method according to an embodiment of the present invention.
[0031] Figure 2 This is a structural diagram of a lightweight cell localization model shown in an embodiment of the present invention.
[0032] Figure 3 This invention illustrates a lightweight cell localization model based on U-Net.
[0033] Figure 4 This is a schematic diagram of a CBAM attention module as shown in an embodiment of the present invention. Detailed Implementation
[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and systems consistent with some aspects of this application as detailed in the appended claims.
[0035] To address the problems existing in current cell localization technologies, this application proposes a lightweight cell localization model based on U-Net, which solves the problems of uneven cell density distribution and bloated models in the current cell localization field, thereby improving the localization performance of cell models.
[0036] Figure 1 This is a flowchart illustrating a lightweight cell localization method according to an exemplary embodiment. The method may include the following steps:
[0037] Step S1: Input the image to be processed into the cell localization model; wherein, the cell localization model is a model obtained after lightweight processing, and differential convolution and attention modules are introduced into the cell localization model;
[0038] Step S2: At the beginning of the initial stage of the cell localization model, the gradient information of the image is enhanced by differential convolution to obtain a feature map containing gradient information.
[0039] Step S3: In each subsequent stage of the cell localization model, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized through the attention module to obtain the optimized feature map.
[0040] Step S4: Obtain cell localization information based on the optimized feature map.
[0041] The proposed solution introduces an attention module into the cell localization model, enabling it to notice densely packed cells in the scene. This significantly enhances the model's localization performance while increasing computational cost slightly. Differential convolution is also introduced into the model to effectively reduce the absolute value difference between pixels, allowing the model to focus more on gradient information between images and thus improving the model's accuracy in recognizing lighter-colored cells. Furthermore, the cell localization model is lightweighted to reduce its computational cost, making it applicable to more low-computing scenarios.
[0042] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.
[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0044] like Figure 2 As shown, this invention proposes a design scheme for a lightweight cell localization model based on U-Net, which specifically includes three parts:
[0045] S100: Differential convolution is introduced, replacing the vanilla convolution in the U-Net encoding part with differential convolution, which enhances the model's ability to recognize lighter-colored cells;
[0046] S200: The attention module is introduced into the U-Net network structure, enabling the model to notice the distribution information of cells in the image and then perform targeted optimization on dense regions.
[0047] S300: Lightweight design, which reduces the weight of the model by pruning the U-Net model and introducing the GhostBottleneck module and grouped convolutions.
[0048] This method can quickly and accurately locate cells in images, thereby further improving the efficiency of medical assistance.
[0049] like Figure 3 As shown below, the solution of this application will be further explained in conjunction with specific application scenarios.
[0050] In some embodiments, the cell localization model is constructed by using a U-Net network, introducing differential convolution at the very front of the feature encoding part of the U-Net network, and introducing an attention module at the end of each stage of the feature encoding part.
[0051] S100: Encoding part of the U-Net model based on differential convolution
[0052] Because cells exhibit varying shades after staining, existing vanilla convolution-based models often overlook lighter-colored cells, thus reducing localization performance. Therefore, introducing differential convolution into the U-Net model effectively reduces the absolute differences between pixels, allowing the model to focus more on gradient information across images, thereby enhancing the model's accuracy in recognizing lighter-colored cells.
[0053] In practical applications, differential convolution is introduced at the very front of the feature encoding part of the U-Net network, including the following steps: replacing the two convolutional layers at the very front of the U-Net network with differential convolution.
[0054] To effectively enhance the gradient information of the image, differential convolution is applied to the feature encoding part at the very front of U-Net. For example... Figure 3 As shown, replacing the first two convolutional layers with this allows the model to ignore the specific color (pixel value) of the cells to some extent and focus more on the gradient information between the cells and the background.
[0055] S200: A U-Net model based on the CBAM (Convolutional Block Attention Module) attention module.
[0056] The attention module is used to: optimize the feature map, send the optimized feature map to the next stage, and laterally jump to the corresponding decoding stage.
[0057] Because cells are often unevenly distributed, existing models do not pay enough attention to densely populated areas, resulting in poor cell localization in these regions. To address this, the CBAM attention module is introduced, enabling the model to notice the distribution information of cells in the image and then perform targeted optimization for densely populated areas.
[0058] In the S200 CBAM attention module, this invention only uses the CBAM attention module to alleviate the problem of uneven density distribution in cells. However, since there are many attention methods, other attention modules can be used instead. Similar methods of adding attention are not substantially different from those in this invention.
[0059] The specific steps are as follows:
[0060] S210, CBAM Principles: Channel Attention, Spatial Attention
[0061] like Figure 4As shown, the CBAM attention mechanism mainly consists of a channel attention module and a spatial attention module. The feature map is first input to the channel attention module, and its output is multiplied by the original feature map for adaptive channel adjustment. Then, it is input to the spatial attention module, and its output is multiplied by the feature map for adaptive spatial adjustment. Finally, the optimized feature map is output. CBAM starts from both channel and spatial domains, introducing two analytical dimensions: channel attention and spatial attention, to achieve a sequential attention structure from channel to space. Channel attention is used to handle the allocation relationship of feature map channels, while spatial attention allows the neural network to focus more on pixel regions in the image that play a decisive role in classification while ignoring irrelevant regions. Simultaneously, the attention allocation across both dimensions enhances the performance improvement effect of the attention mechanism on the model.
[0062] S220: U-Net based on CBAM
[0063] like Figure 3 As shown, CBAM modules are introduced into the encoding part of U-Net. Specifically, they are introduced into each end of the encoding part (except the first one), for a total of four CBAM modules. Each module optimizes the features, enabling the model to notice the distribution information of cells in the image. Finally, the feature maps are fed into the next stage and laterally skipped to the corresponding decoding stage.
[0064] S300: Lightweight Model Processing: Pruning U-Net and Introducing Grouped Convolutions and Ghost Bottleneck Modules
[0065] Considering that the number of parameters and inference speed of existing cell localization models are insufficient for practical applications, the model is lightweighted while maintaining its localization accuracy. The specific steps are as follows:
[0066] S310, Ghost Bottleneck module
[0067] First, we introduce the Ghost module, followed by the Ghost Bottleneck module based on Ghost. A well-trained deep neural network typically contains rich, even redundant, feature maps to ensure a comprehensive understanding of the input data. The Ghost module generates rich feature maps through clever and inexpensive operations, rather than complex convolution operations, thus saving significant computational resources and parameters.
[0068] Specifically, such as Figure 3As shown, the Ghost module decomposes a regular convolutional layer into two parts. The first part is a normal convolution with reduced channels. Then, a series of simple linear operations are applied to the feature map output from the first part to generate more feature maps. Compared to a regular convolutional neural network, the total number of parameters and computational complexity required in this Ghost module are reduced without changing the output feature map size. Building on the Ghost module, researchers further developed the Ghostbottleneck lightweight module. This module contains two Ghost modules and an SE channel attention module. The first Ghost module varies the number of channels in the features while generating rich feature maps. The resulting feature maps are then fed into the SE channel attention module for adaptive adjustment of the channel features. Finally, the features are input into the second Ghost module and added to the input.
[0069] S320, cropping U-Net and introducing grouped convolutions
[0070] The original U-Net model had 34.53M parameters and 65.47 Gflops of computation, resulting in slow inference speed in practical applications. First, the number of channels was pruned. The original number of channels for each stage was [3, 64, 128, 256, 512, 1024, 512, 256, 128, 64, 1], as shown in the attached diagram. Figure 3 As shown, after cropping, it becomes [3,32,64,128,256,512,256,128,64,32,1]. Next, grouped convolution is introduced into the U-Net convolution operation. Since grouped convolution divides the feature map into multiple groups according to channels, and then performs convolution on each group, the number of parameters is reduced. Then, the Ghost Bottleneck module is introduced into the U-Net network. Finally, differential convolution and CBAM are introduced into the network, improving the model's localization performance with almost no increase in computation. The specific changes in parameter and computational costs are shown in Table 1.
[0071] Table 1 shows the changes in the number of parameters and computational cost of the lightweight model based on U-Net.
[0072]
[0073] In summary, the proposed solution introduces the CBAM attention module into the U-Net network, enabling the U-Net model to notice densely packed cells in the scene, effectively enhancing the model's localization performance while increasing computational cost slightly. Furthermore, the U-Net network is lightweighted through grouped convolutions and the introduction of the Ghost Bottleneck module, reducing the computational cost of the U-Net model and allowing it to be applied to more low-computing scenarios.
[0074] In one embodiment, this application also provides a lightweight cell localization system, including: an input module and a cell localization model; the cell localization model is a model obtained after lightweight processing, and differential convolution and attention modules are introduced into the cell localization model.
[0075] The input module is used to input the image to be processed into the cell localization model. The cell localization model is used in the initial stage to enhance the gradient information of the image through differential convolution, obtaining a feature map containing gradient information; in subsequent stages, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized through an attention module to obtain an optimized feature map; cell localization information is obtained based on the optimized feature map.
[0076] Regarding the system in the above embodiments, the specific steps for each module to perform operations have been described in detail in the embodiments related to the method, and will not be elaborated further here. Each module in the above lightweight cell localization system can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0077] In one embodiment, this application also provides a computer device. The computer device includes a processor and a memory connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the computer program is executed by the processor, it implements a lightweight cell localization method: an image to be processed is input into a cell localization model; wherein the cell localization model is a model obtained after lightweight processing, and differential convolution and an attention module are introduced into the cell localization model; at the initial stage of the cell localization model, the gradient information of the image is enhanced by differential convolution to obtain a feature map containing gradient information; in subsequent stages of the cell localization model, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized by the attention module to obtain an optimized feature map; cell localization information is obtained based on the optimized feature map.
[0078] In one embodiment, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a lightweight cell localization method: an image to be processed is input into a cell localization model; wherein the cell localization model is a model obtained after lightweight processing, and differential convolution and an attention module are introduced into the cell localization model; at the initial stage of the cell localization model, differential convolution is used to enhance the gradient information of the image to obtain a feature map containing gradient information; in subsequent stages of the cell localization model, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized through the attention module to obtain an optimized feature map; cell localization information is obtained based on the optimized feature map.
[0079] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0080] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.
[0081] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.
[0082] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0083] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0085] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0086] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0087] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.
Claims
1. A lightweight cell localization method, characterized in that, include: The image to be processed is input into a cell localization model; wherein, the cell localization model is a model obtained after lightweight processing, and differential convolution and attention modules are introduced into the cell localization model; wherein, the lightweight processing of the cell localization model includes the following steps: cropping the number of channels for convolution operations in each stage of the cell localization model; introducing grouped convolution into the convolution operations of the cell localization model; and introducing the Ghost Bottleneck module into the cell localization model; At the beginning of the initial stage of the cell localization model, the gradient information of the image is enhanced by differential convolution to obtain a feature map containing gradient information. In subsequent stages of the cell localization model, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized through an attention module to obtain an optimized feature map. Cell localization information is obtained based on the optimized feature map.
2. The method according to claim 1, characterized in that, The method for constructing the cell localization model is as follows: The U-Net network is used, with differential convolution introduced at the very beginning of the feature encoding part and attention modules introduced at the end of each stage of the feature encoding part.
3. The method according to claim 2, characterized in that, Differential convolution is introduced at the very front of the feature encoding part of the U-Net network, including the following steps: Replace the two frontmost convolutional layers of the U-Net network with differential convolutions.
4. The method according to claim 2, characterized in that, The attention module is used to: optimize the feature map, send the optimized feature map to the next stage, and laterally jump to the corresponding decoding stage.
5. The method according to claim 1, characterized in that, The number of channels performing convolution operations at each stage in the cell localization model is pruned, including: The number of channels in each stage of the cell localization model was cropped to [3, 32, 64, 128, 256, 512, 256, 128, 64, 32, 1].
6. The method according to claim 2, characterized in that, Introducing grouped convolution into the convolution operation of the U-Net network includes the following steps: The feature map is divided into multiple groups according to channels, and then convolution is performed on each group.
7. A lightweight cell positioning system, characterized in that, include: The system includes an input module and a cell localization model. The cell localization model is a lightweighted model that incorporates differential convolution and an attention module. The lightweighting process includes the following steps: pruning the number of channels used in convolution operations at each stage of the cell localization model; introducing grouped convolution into the convolution operations of the cell localization model; and introducing the Ghost Bottleneck module into the cell localization model. The input module is used to input the image to be processed into the cell localization model; The cell localization model is used in the initial stage to enhance the gradient information of the image through differential convolution to obtain a feature map containing gradient information; in subsequent stages, multi-channel convolution is performed on the feature map containing gradient information, and the feature map is adaptively optimized through an attention module to obtain an optimized feature map; cell localization information is obtained based on the optimized feature map.
8. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program in the memory to implement the operational steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the operational steps of the method according to any one of claims 1 to 7.