Signal point identification method and related product

Abstract

The invention provides a signal point identification method and a related product. According to a specific embodiment of the method, instance segmentation is performed on a cell nucleus image based on a cell instance segmentation model, and a cell segmentation result of each cell in the cell nucleus image is obtained; based on a fusion feature extraction model, carrying out image feature extraction on the fluorescence signal point image of each channel to obtain a fusion feature map; then, based on the cell segmentation result of each cell, extracting a cell region fusion feature map of each cell from the fusion feature map; then, based on a cell region mask feature extraction model, obtaining a masked cell region fusion feature map of each cell; and finally, based on a signal point identification model, performing signal point identification on the masked cell region fusion feature map of each cell to obtain a signal point identification result of the corresponding cell in each channel. Therefore, end-to-end cell segmentation and signal point identification can be realized, cell segmentation and signal point identification can be realized based on the cell nucleus image and the multi-channel fluorescence signal point image, cell segmentation and signal point identification can be completed at the same time with fewer model parameters, the calculation amount is saved, and the algorithm processing time is accelerated.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of machine learning technology, and in particular to a signal point recognition method and related products. Background Art

[0002] Fluorescent in situ hybridization (FISH) is a cytogenetic technique that can be used to detect and locate nucleic acids. Fluorescently labeled nucleic acid probes only hybridize with nucleic acids with high similarity and can be used to locate genes on chromosomes, or to label ribosomal RNA in bacteria or archaea of ​​different classifications in molecular ecology. FISH microscopy based on multiple fluorescent markers can specifically detect diseases with high sensitivity. For example, circulating genetically abnormal cells (CAC) in lung cancer can be detected and judged one by one in the field of view through four-channel fluorescent signal point images and Dapi images. Traditional manual annotation methods are time-consuming and labor-intensive, so various computer vision methods have emerged to automatically identify CAC cells.

[0003] However, the current practice is to segment cells based on the Dapi image, locate each cell in the four channel signal point images and intercept the signal point images of the corresponding cells in the corresponding channels, then perform signal point recognition on the intercepted signal point images of each channel to obtain the signal point recognition results of the corresponding channels, and finally determine whether the cell is a CAC cell based on the signal point recognition results of each cell in the signal point images of each channel. The important part of this process is to obtain the signal point recognition results of each cell in the signal point images of each channel. At present, the above process is divided into two stages: cell segmentation and signal point recognition. However, the above two-stage method needs to locate each cell in the signal point images of the four channels, and then set the corresponding recognition model for signal point recognition according to the signal point images of each cell in each channel, which leads to more model parameters and more storage space, as well as a large amount of calculation and low calculation efficiency. Summary of the invention

[0004] The embodiments of the present disclosure provide a signal point identification method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0005] In a first aspect, an embodiment of the present disclosure provides a signal point identification method, the method comprising:

[0006] Acquire the cell nucleus image and the fluorescence signal point image of each channel taken in the same field of view;

[0007] Based on the cell instance segmentation model, instance segmentation is performed on the cell nucleus image to obtain a cell segmentation result of each cell in the cell nucleus image, wherein the cell segmentation result includes a cell image mask and cell position information;

[0008] Based on the fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel to obtain a fusion feature map;

[0009] Based on the cell segmentation results of each cell in the cell nucleus image, extracting a cell region fusion feature map of each cell in the fusion feature map;

[0010] Based on the cell region mask feature extraction model, the cell image mask and the cell region fusion feature map of each cell are fused and extracted to obtain the masked cell region fusion feature map of the corresponding cell;

[0011] Based on the signal point recognition model, signal point recognition is performed on the masked cell area fusion feature map of each cell to obtain the signal point recognition results of the corresponding cells in each channel.

[0012] In some optional implementations, the fusion feature extraction model includes a primary fusion feature extraction model and a multi-scale feature extraction model; and

[0013] The method of extracting image features of the fluorescence signal point images of each channel based on the fusion feature extraction model to obtain a fusion feature map includes:

[0014] Based on the primary fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel to obtain a preliminary fusion feature map;

[0015] Based on a multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain the fusion feature map.

[0016] In some optional embodiments, the image feature extraction of the fluorescence signal point image of each channel based on the primary fusion feature extraction model to obtain a preliminary fusion feature map includes:

[0017] Based on multiple continuous depth-wise separable convolutional layers, image feature extraction is performed on the fluorescence signal point image of each channel in turn, and channel image features of each channel output by the last depth-wise separable convolutional layer are obtained, wherein the convolution kernels in each depth-wise separable convolutional layer are divided into multiple groups, the number of groups of convolution kernels is the same as the number of channels corresponding to each of the fluorescence signal point images, each group of convolution kernels is used to extract image features from the fluorescence signal point image of the corresponding channel, and the number of convolution kernels of the multiple continuous depth-wise separable convolutional layers increases in turn with the increase of convolutional layers;

[0018] Feature fusion is performed on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map.

[0019] In some optional implementations, the step of performing feature fusion on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map includes:

[0020] The concatenation result of the channel image features of each channel is convolved to obtain the preliminary fusion feature map.

[0021] In some optional implementations, the multi-scale feature extraction model includes a multi-scale feature extraction network and a feature pyramid network; and

[0022] The step of performing multi-scale feature extraction on the preliminary fusion feature map based on the multi-scale feature extraction model to obtain the fusion feature map includes:

[0023] Based on a multi-scale feature extraction network, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale down-sampling feature map;

[0024] Inputting the multi-scale downsampled feature maps into a feature pyramid network from top to bottom in order of scale from small to large;

[0025] The output result of the bottom layer of the feature pyramid network is determined as the fused feature map.

[0026] In some optional embodiments, the cell region mask feature extraction model is based on which the cell image mask and the cell region fusion feature map of each cell are subjected to feature fusion and extraction to obtain the masked cell region fusion feature map of the corresponding cell, including:

[0027] Based on the cell segmentation result of each cell, the cell image mask of the cell is applied to the cell region fusion feature map of the cell to obtain a preliminary masked feature map of the cell, and feature fusion is performed on the preliminary masked feature map of the cell to obtain a masked cell region fusion feature map of the cell.

[0028] In some optional implementations, the signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and

[0029] Based on the signal point recognition model, the signal point recognition is performed on the masked cell area fusion feature map of each cell to obtain the signal point recognition results of the corresponding cell in each channel, including:

[0030] Based on the signal point area information map extraction model, the signal point area information map is extracted from the masked cell area fusion feature map of each cell to obtain the signal point area information map of the corresponding cell in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescent signal point image;

[0031] Based on the signal point heat map extraction model, a signal point heat map is extracted from the masked cell area fusion feature map of each cell to obtain a signal point heat map of the corresponding cell in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map;

[0032] Based on the signal point area information map and signal point heat map of each cell in each channel, the signal point identification results of the corresponding cells in each channel are determined.

[0033] In some optional embodiments, the signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0034] In some optional embodiments, the first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

[0035] In some optional implementations, the signal point area information graph extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0036] In some optional embodiments, the signal point width extraction model includes a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model includes a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model includes a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model includes a sixth image feature extraction network and a convolutional layer connected in sequence.

[0037] In some optional embodiments, the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are pre-trained through the following training steps:

[0038] Based on the sample cell nuclear image and the labeled cell segmentation result of each cell in the sample cell nuclear image, the cell instance segmentation model is trained, and the labeled cell segmentation result includes a labeled cell image mask and labeled cell position information;

[0039] Based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, as well as the labeled cell segmentation results of each cell in the sample cell nuclear image and the labeled signal point recognition results in each channel, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained to obtain.

[0040] In some optional embodiments, the training of the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, the annotated cell segmentation result of each cell in the sample cell nuclear image and the annotated signal point recognition result in each channel includes:

[0041] Based on the initial fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a first sample fusion feature map;

[0042] Extracting a first sample cell region fusion feature map of each cell in the first sample fusion feature map based on the labeled cell position information of each cell in the sample cell nuclear image;

[0043] Based on the initial cell region mask feature extraction model, feature fusion and extraction are performed on the labeled cell image mask of each cell in the sample cell nucleus image and the first sample cell region fusion feature map to obtain the first sample masked cell region fusion feature map of the corresponding cell;

[0044] Based on the initial signal point recognition model, signal point recognition is performed on the first sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain the first actual signal point recognition result of the corresponding cell in each channel;

[0045] Based on the difference between the first actual signal point recognition result and the labeled signal point recognition result of each cell in the sample cell nucleus image in each channel, the model parameters of the initial fusion feature extraction model, the initial cell area mask feature extraction model and the initial signal point recognition model are adjusted to obtain the trained fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0046] In some optional implementations, the training step further includes:

[0047] Based on the sample cell nuclear image, the labeled cell segmentation result of each cell in the sample cell nuclear image and the labeled signal point recognition result in each channel, and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image, the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained.

[0048] In some optional embodiments, the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are trained based on the sample cell nuclear image, the annotated cell segmentation result of each cell in the sample cell nuclear image and the annotated signal point recognition result in each channel, and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image, including:

[0049] Based on the cell instance segmentation model, the sample cell nuclear image is instance segmented to obtain actual cell segmentation results of each cell in the sample cell nuclear image, wherein the actual cell segmentation results include actual cell image masks and actual cell position information;

[0050] Based on the fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a second sample fusion feature map;

[0051] Extracting a second sample cell region fusion feature map of each cell in the second sample fusion feature map based on actual cell position information of each cell in the sample cell nucleus image;

[0052] Based on the cell region mask feature extraction model, feature fusion and extraction are performed on the actual cell image mask of each cell in the sample cell nucleus image and the second sample cell region fusion feature map to obtain the second sample masked cell region fusion feature map of the corresponding cell;

[0053] Based on the signal point recognition model, signal point recognition is performed on the second sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain a second actual signal point recognition result of the corresponding cell in each channel;

[0054] Based on the difference between the second actual cell segmentation result and the labeled cell segmentation result of each cell in the sample cell nucleus image, and the difference between the second actual signal point recognition result and the labeled signal point recognition result in each channel, the model parameters of the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are adjusted to obtain the trained cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0055] In a second aspect, an embodiment of the present disclosure provides a signal point identification device, the device comprising:

[0056] An acquisition module is configured to acquire a cell nucleus image and a fluorescence signal point image of each channel captured in the same field of view;

[0057] A cell segmentation module is configured to perform instance segmentation on the cell nucleus image based on a cell instance segmentation model to obtain a cell segmentation result of each cell in the cell nucleus image, wherein the cell segmentation result includes a cell image mask and cell position information;

[0058] The signal point feature fusion module is configured to extract image features of the fluorescence signal point images of each channel based on the fusion feature extraction model to obtain a fusion feature map;

[0059] A regional signal point feature extraction module is configured to extract a cell region fusion feature map of each cell in the fusion feature map based on the cell segmentation results of each cell in the cell nucleus image;

[0060] The mask processing module is configured to perform feature fusion and extraction on the cell image mask and the cell region fusion feature map of each cell based on the cell region mask feature extraction model to obtain the masked cell region fusion feature map of the corresponding cell;

[0061] The signal point recognition module is configured to perform signal point recognition on the masked cell area fusion feature map of each cell based on the signal point recognition model to obtain the signal point recognition results of the corresponding cell in each channel.

[0062] In some optional implementations, the fusion feature extraction model includes a primary fusion feature extraction model and a multi-scale feature extraction model; and

[0063] The signal point feature fusion module includes:

[0064] The feature fusion unit is configured to extract image features from the fluorescence signal point images of each channel based on the primary fusion feature extraction model to obtain a preliminary fusion feature map;

[0065] The multi-scale feature extraction unit is configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain the fusion feature map.

[0066] In some optional implementations, the feature fusion unit is further configured to:

[0067] Based on multiple continuous depth-wise separable convolutional layers, image feature extraction is performed on the fluorescence signal point image of each channel in turn, and channel image features of each channel output by the last depth-wise separable convolutional layer are obtained, wherein the convolution kernels in each depth-wise separable convolutional layer are divided into multiple groups, the number of groups of convolution kernels is the same as the number of channels corresponding to each of the fluorescence signal point images, each group of convolution kernels is used to extract image features from the fluorescence signal point image of the corresponding channel, and the number of convolution kernels of the multiple continuous depth-wise separable convolutional layers increases in turn with the increase of convolutional layers;

[0068] Feature fusion is performed on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map.

[0069] In some optional implementations, the step of performing feature fusion on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map includes:

[0070] The concatenation result of the channel image features of each channel is convolved to obtain the preliminary fusion feature map.

[0071] In some optional implementations, the multi-scale feature extraction model includes a multi-scale feature extraction network and a feature pyramid network; and

[0072] The multi-scale feature extraction unit is further configured as follows:

[0073] Based on a multi-scale feature extraction network, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale down-sampling feature map;

[0074] Inputting the multi-scale downsampled feature maps into a feature pyramid network from top to bottom in order of scale from small to large;

[0075] The output result of the bottom layer of the feature pyramid network is determined as the fused feature map.

[0076] In some optional implementations, the mask processing module is further configured to:

[0077] Based on the cell segmentation result of each cell, the cell image mask of the cell is applied to the cell region fusion feature map of the cell to obtain a preliminary masked feature map of the cell, and feature fusion is performed on the preliminary masked feature map of the cell to obtain a masked cell region fusion feature map of the cell.

[0078] In some optional implementations, the signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and

[0079] The signal point identification module is further configured to:

[0080] Based on the signal point area information map extraction model, the signal point area information map is extracted from the masked cell area fusion feature map of each cell to obtain the signal point area information map of the corresponding cell in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescent signal point image;

[0081] Based on the signal point heat map extraction model, a signal point heat map is extracted from the masked cell area fusion feature map of each cell to obtain a signal point heat map of the corresponding cell in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map;

[0082] Based on the signal point area information map and signal point heat map of each cell in each channel, the signal point identification results of the corresponding cells in each channel are determined.

[0083] In some optional embodiments, the signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0084] In some optional embodiments, the first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

[0085] In some optional implementations, the signal point area information graph extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0086] In some optional embodiments, the signal point width extraction model includes a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model includes a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model includes a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model includes a sixth image feature extraction network and a convolutional layer connected in sequence.

[0087] In some optional embodiments, the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are pre-trained through the following training steps:

[0088] Based on the sample cell nuclear image and the labeled cell segmentation result of each cell in the sample cell nuclear image, the cell instance segmentation model is trained, and the labeled cell segmentation result includes a labeled cell image mask and labeled cell position information;

[0089] Based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, as well as the labeled cell segmentation results of each cell in the sample cell nuclear image and the labeled signal point recognition results in each channel, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained to obtain.

[0090] In some optional embodiments, the training of the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, the annotated cell segmentation result of each cell in the sample cell nuclear image and the annotated signal point recognition result in each channel includes:

[0091] Based on the initial fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a first sample fusion feature map;

[0092] Extracting a first sample cell region fusion feature map of each cell in the first sample fusion feature map based on the labeled cell position information of each cell in the sample cell nuclear image;

[0093] Based on the initial cell region mask feature extraction model, feature fusion and extraction are performed on the labeled cell image mask of each cell in the sample cell nucleus image and the first sample cell region fusion feature map to obtain the first sample masked cell region fusion feature map of the corresponding cell;

[0094] Based on the initial signal point recognition model, signal point recognition is performed on the first sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain the first actual signal point recognition result of the corresponding cell in each channel;

[0095] Based on the difference between the first actual signal point recognition result and the labeled signal point recognition result of each cell in the sample cell nucleus image in each channel, the model parameters of the initial fusion feature extraction model, the initial cell area mask feature extraction model and the initial signal point recognition model are adjusted to obtain the trained fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0096] In some optional implementations, the training step further includes:

[0097] Based on the sample cell nuclear image, the labeled cell segmentation result of each cell in the sample cell nuclear image and the labeled signal point recognition result in each channel, and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image, the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained.

[0098] In some optional embodiments, the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are trained based on the sample cell nuclear image, the annotated cell segmentation result of each cell in the sample cell nuclear image and the annotated signal point recognition result in each channel, and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image, including:

[0099] Based on the cell instance segmentation model, the sample cell nuclear image is instance segmented to obtain actual cell segmentation results of each cell in the sample cell nuclear image, wherein the actual cell segmentation results include actual cell image masks and actual cell position information;

[0100] Based on the fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a second sample fusion feature map;

[0101] Extracting a second sample cell region fusion feature map of each cell in the second sample fusion feature map based on actual cell position information of each cell in the sample cell nucleus image;

[0102] Based on the cell region mask feature extraction model, feature fusion and extraction are performed on the actual cell image mask of each cell in the sample cell nucleus image and the second sample cell region fusion feature map to obtain the second sample masked cell region fusion feature map of the corresponding cell;

[0103] Based on the signal point recognition model, signal point recognition is performed on the second sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain a second actual signal point recognition result of the corresponding cell in each channel;

[0104] Based on the difference between the second actual cell segmentation result and the labeled cell segmentation result of each cell in the sample cell nucleus image, and the difference between the second actual signal point recognition result and the labeled signal point recognition result in each channel, the model parameters of the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are adjusted to obtain the trained cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0105] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0106] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner in the first aspect.

[0107] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which implements the method described in any implementation manner in the first aspect when the computer program / instruction is executed by a processor.

[0108] In order to solve the problem that the current two-stage method for identifying signal points in a fluorescent signal point image requires many model parameters and thus has low computational efficiency, the signal point identification method, device, electronic device, storage medium and computer program product provided by the embodiments of the present disclosure first perform instance segmentation on the cell nucleus image based on the cell instance segmentation model to obtain the cell segmentation results of each cell in the cell nucleus image; then perform image feature extraction on the fluorescent signal point image of each channel based on the fusion feature extraction model to obtain a fusion feature map; then, based on the cell segmentation results of each cell, extract the cell region fusion feature map of each cell in the fusion feature map; then, based on the cell region mask feature extraction model, perform feature fusion and extraction on the cell image mask and the cell region fusion feature map of each cell to obtain the masked cell region fusion feature map of the corresponding cell; finally, based on the signal point recognition model, perform signal point recognition on the masked cell region fusion feature map of each cell to obtain the signal point recognition results of the corresponding cell in each channel. Thus, only the cell nucleus image and the fluorescent signal point image of each channel taken under the same field of view need to be input, and the cell position of each cell in the cell nucleus image, the cell image mask and the signal point recognition results of the corresponding cell in each channel can be output. This enables end-to-end cell segmentation and signal point recognition based on cell nucleus images and multi-channel fluorescence signal point images, and can simultaneously complete cell segmentation and signal point recognition with fewer model parameters, saving computational effort and improving computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0110] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0111] Figure 2A is a flow chart of an embodiment of a signal point identification method according to the present disclosure;

[0112] Figure 2B is a decomposed flow chart of one embodiment of step 203 according to the present disclosure;

[0113] Figure 2C is a decomposed flow chart of one embodiment of step 2031 according to the present disclosure;

[0114] Figure 2D is a decomposed flow chart of one embodiment of step 2032 according to the present disclosure;

[0115] Figure 2E is a decomposed flow chart of one embodiment of step 206 according to the present disclosure;

[0116] Figure 3 is a schematic diagram of the structure of three-layer depth-separable convolutional layers according to the present disclosure;

[0117] Figure 4 is a schematic diagram of the operation process of an embodiment of inputting a multi-scale down-sampled feature map output by a multi-scale feature extraction network into a feature pyramid network according to the present disclosure;

[0118] Figure 5A and Figure 5B is a schematic diagram of the operation process of step 205 according to the present disclosure;

[0119] Fig. 6A is a flow chart of one embodiment of the training step 600 according to the present disclosure;

[0120] Figure 6B is a decomposed flow chart of one embodiment of step 602 according to the present disclosure;

[0121] Figure 6C is a decomposed flow chart of one embodiment of step 603 according to the present disclosure;

[0122] Figure 7 is a schematic diagram of an application scenario of an embodiment of the signal point identification method disclosed in the present invention;

[0123] Figure 8 is a structural schematic diagram of an embodiment of a signal point identification device according to the present disclosure;

[0124] Fig. 9 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0125] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0126] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0127] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the signal point identification method, apparatus, electronic device, and storage medium of the present disclosure can be applied.

[0128] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0129] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as signal point recognition applications, short video social applications, audio and video conferencing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0130] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices (e.g., keyboard, mouse, touch screen, microphone, camera, etc.) and information output devices (e.g., display screen, speaker, etc.), including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, used to provide signal point identification services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0131] In some cases, the signal point identification method provided by the present disclosure may be executed by the terminal devices 101, 102, 103, and accordingly, the signal point identification apparatus may be disposed in the terminal devices 101, 102, 103. In this case, the system architecture 100 may not include the server 105.

[0132] In some cases, the signal point recognition method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining the cell nucleus image and the fluorescence signal point image of each channel taken under the same field of view" can be performed by the terminal devices 101, 102, 103, and the steps of "performing instance segmentation on the cell nucleus image based on the cell instance segmentation model to obtain the cell segmentation results of each cell in the cell nucleus image" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the signal point recognition device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0133] In some cases, the signal point identification method provided in the present disclosure may be executed by the server 105 , and accordingly, the signal point identification device may also be set in the server 105 . In this case, the system architecture 100 may not include the terminal devices 101 , 102 , and 103 .

[0134] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0135] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0136] Continue to refer Figure 2A , which shows a process 200 of an embodiment of a signal point identification method according to the present disclosure, the signal point identification method comprises the following steps:

[0137] Step 201 : acquiring a cell nucleus image and a fluorescence signal point image of each channel captured in the same field of view.

[0138] In this embodiment, the execution subject of the signal point identification method (for example Figure 1 The terminal device shown in the figure can obtain the cell nucleus image and the fluorescence signal point image of each channel taken in the same field of view locally or remotely from other electronic devices connected to the above-mentioned execution subject network.

[0139] Here, the cell nucleus image (for example, a Dapi image) and the fluorescence signal point image of each channel (for example, a fluorescence image taken under the four channels of Gold, Red, Blue and Green) taken by the microscope in the same field of view can be obtained, wherein the fluorescence signal point image of each channel includes the signal point imaging inside each cell.

[0140] The cell nucleus image may be an image captured under a microscope after staining the cell nuclei in the sample, for example, a Dapi image captured after staining the cell nuclei with DAPI fluorescent dye.

[0141] The fluorescent signal point images of each channel are images taken by using the fluorescent labeled probes corresponding to the corresponding channels to detect multiple DNA sequences or chromosomes in the sample. Each fluorescent dye corresponds to a specific fluorescent labeled probe, so that the expression and location information of multiple genes or chromosomes can be analyzed simultaneously in one sample. For example, fluorescent dyes such as FITC, TRITC, Cy5 and Cy7 can be used, and each dye corresponds to a channel, which is used to label different nucleic acid sequences.

[0142] In some embodiments, the cell nucleus image and the fluorescence signal point image of each channel may be grayscale images.

[0143] Step 202: Based on the cell instance segmentation model, instance segmentation is performed on the cell nucleus image to obtain cell segmentation results of each cell in the cell nucleus image.

[0144] Here, the executor of the signal point recognition method can perform instance segmentation on the cell nucleus image obtained in step 201 based on the cell instance segmentation model to obtain the cell segmentation results of each cell in the cell nucleus image. Specifically, the cell nucleus image obtained in step 201 can be input into a pre-trained cell instance segmentation model to obtain the cell segmentation results of each cell in the cell nucleus image.

[0145] Here, the cell segmentation result may include a cell image mask and cell position information. Among them, the cell image mask is used to characterize whether the pixel point at the corresponding position of the cell nucleus image is a cell pixel point or a background pixel point. The cell position information is used to characterize the position or area (e.g., a rectangular area) of the cell in the cell nucleus image. As an example, the cell position information may include the four-point vertex coordinates of the rectangular area where the cell is located, or a vertex coordinate and the width and height of the rectangular area.

[0146] Here, the cell instance segmentation model is used to characterize the correspondence between the cell nucleus image and the cell segmentation result. In practice, the cell instance segmentation model can adopt various existing or future developed instance segmentation models, and the present disclosure does not make specific limitations on this.

[0147] As an example, the cell instance segmentation model can be HoVner-Net (HoVer-Net is a deep learning model specially designed for simultaneous segmentation and classification of cell nuclei in multiple histological images). It is understandable that the cell instance segmentation model can also be obtained by using other instance segmentation models after training with sample cell nucleus images and corresponding annotated cell segmentation results.

[0148] Step 203: Based on the fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel to obtain a fusion feature map.

[0149] In this embodiment, the executor of the signal point recognition method may extract image features from the fluorescence signal point images of each channel acquired in step 201 based on a fusion feature extraction model to obtain a fusion feature map.

[0150] Here, the fusion feature extraction model is used to extract image features from the fluorescence signal point image of each channel, and then fuse the image features extracted from the fluorescence signal point images of each channel to obtain a fusion feature map. Thus, the image features of the fluorescence signal point images of each channel are learned in the fusion feature map.

[0151] In some optional implementations, the fusion feature extraction model may include a primary fusion feature extraction model and a multi-scale feature extraction model. Accordingly, step 203 may include: Figure 2B The following steps 2031 and 2032 are shown:

[0152] Step 2031, based on the primary fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel to obtain a preliminary fusion feature map.

[0153] Here, the primary fusion feature extraction model is used to extract image features from the fluorescence signal point image of each channel, and then fuse the image features extracted from the fluorescence signal point images of each channel to obtain a preliminary fusion feature map.

[0154] For example, the primary fusion feature extraction model may include N channel feature extraction models, where N is the number of fluorescence signal point images, or the number of channels. Each channel corresponds to a channel feature extraction model. Furthermore, the fluorescence signal point image of each channel may be input into the corresponding channel feature extraction model to obtain the image features of the channel, and then the image features of each channel may be fused to obtain a preliminary fusion feature map.

[0155] Optionally, step 2031 may also include: Figure 2C The following steps 20311 and 20312 are shown:

[0156] Step 20311, based on multiple consecutive depth-wise separable convolutional layers, image features of the fluorescence signal point images of each channel are extracted in sequence to obtain channel image features of each channel output by the last depth-wise separable convolutional layer.

[0157] Here, the convolution kernels in each depth-separable convolution layer are divided into multiple groups. The number of convolution kernel groups is the same as the number of channels corresponding to each fluorescent signal point image. Each group of convolution kernels is used to extract image features of the fluorescent signal point image of the corresponding channel.

[0158] For example, Figure 3 As shown in the figure, when there are four channels of fluorescence signal point images, there are three layers of depth-separable convolution layers Conv1, Conv2 and Conv3. The convolution kernels in each depth-separable convolution layer are divided into four groups, corresponding to the fluorescence signal point images of four different channels. For example, the fluorescence signal point images of the Red channel, Green channel, Gold channel and Aqua channel correspond to the front view, respectively. Figure 3 The output of each group of convolution kernels in the third layer of depth-separable convolution layer is the channel image feature of the corresponding channel. Figure 3 As shown, features M1, M2, M3 and M4 are channel image features of the Red channel, Green channel, Gold channel and Aqua channel respectively.

[0159] Here, the number of convolution kernels of multiple consecutive depthwise separable convolution layers increases with the number of convolution layers. For example, the number of convolution kernels in the second convolution layer is greater than the number of convolution kernels in the first convolution layer, which ensures that the fluorescence signal point images of each channel have sufficient feature extraction capabilities.

[0160] Step 20312, perform feature fusion on the splicing results of the channel image features of each channel to obtain a preliminary fusion feature map.

[0161] Here, various feature fusion methods can be used to perform feature fusion on the splicing results of the channel image features of each channel to obtain a preliminary fusion feature map. For example, the splicing results of the channel image features of each channel can be directly used as the preliminary fusion feature map.

[0162] Optionally, convolution may be performed on the concatenated results of the channel image features of each channel to obtain a preliminary fused feature map.

[0163] For example, Figure 3 As shown in FIG. 1 , after convolution of the concatenation result M' of features M1, M2, M3 and M4, a preliminary fused feature map M" is obtained.

[0164] Step 2032: Based on the multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a fusion feature map.

[0165] The preliminary fusion feature map extracted in step 2031 (e.g., Figure 3 The preliminary fusion feature map M” shown in FIG only completes the preliminary feature extraction of the fluorescence signal point image of each channel. In order to improve the accuracy of subsequent signal point recognition, more complex features need to be learned. Therefore, the preliminary fusion feature map (for example, Figure 3 The preliminary fused feature map M” shown in FIG. 1 is used to perform multi-scale feature extraction to obtain a fused feature map.

[0166] Here, the multi-scale feature extraction model is used to extract complex features from multiple scales. For example, the multi-scale feature extraction model can be various backbone networks, such as ResNet, VGG, Inception, etc.

[0167] In some optional implementations, the multi-scale feature extraction model may include a multi-scale feature extraction network and a feature pyramid network. Accordingly, step 2032 may include: Figure 2D The following steps 20321 to 20323 are shown:

[0168] Step 20321, based on the multi-scale feature extraction network, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale down-sampling feature map.

[0169] Here, the multi-scale feature extraction network can be any neural network for multi-scale feature extraction, for example, it can be Swin Transformer V2 T. The multi-scale down-sampled feature map may include at least two different scales and feature dimensions greater than the initial fusion feature map (for example, Figure 3 The preliminary fused feature map M” shown in ) is a feature map with lower dimensionality. For example, the multi-scale down-sampled feature map may include four feature maps F1, F2, F3 and F4 with four feature scales from small to large.

[0170] Step 20322, the multi-scale down-sampled feature maps are input into the feature pyramid network from top to bottom in the order of scale from small to large.

[0171] Here, Feature Pyramid Networks (FPN) is used to perform feature fusion on multi-scale downsampled feature maps. The multi-scale downsampled feature maps are input into the feature pyramid network from top to bottom in the order of scale from small to large, and feature fusion is performed on the previous features starting from the second top-down layer of the feature pyramid network.

[0172] For ease of understanding, refer to Figure 4 .like Figure 4 As shown, the multi-scale down-sampled feature map output by the multi-scale feature extraction network may include four feature maps F1, F2, F3 and F4 with feature scales from small to large.

[0173] The specific operation process of inputting the feature pyramid network in step 20322 is as follows:

[0174] (1) In the first layer, the feature map obtained by performing a 1*1 convolution operation on F1 is used as the feature map F1' output from the first top-down layer of the feature pyramid network: that is, F1'=Conv(F1), where Conv represents a 1*1 convolution operation.

[0175] (2) In the second layer, the feature map obtained by performing a 1*1 convolution operation on F2 is added to the feature map with the same dimension as F2 obtained by upsampling F1', which is used as the feature map F2' output from the second top-down layer of the feature pyramid network: that is, F2'=Conv(F2)+up(F1'), where Conv() represents a 1*1 convolution operation and up() represents an upsampling operation.

[0176] (3) In the third layer, the feature map obtained by performing a 1*1 convolution operation on F3 is added to the feature map with the same dimension as F3 obtained by upsampling F2', which is used as the feature map F3' output from the third layer from top to bottom of the feature pyramid network: that is, F3'=Conv(F3)+up(F2').

[0177] (2) In the fourth layer, the feature map obtained by performing a 1*1 convolution operation on F4 is added to the feature map with the same dimension as F4 obtained by upsampling F3', which is used as the feature map F4' output from the fourth layer from top to bottom of the feature pyramid network: that is, F4'=Conv(F4)+up(F3').

[0178] Step 20323, determine the output result of the bottom layer of the feature pyramid network as the fused feature map.

[0179] According to the above example, F4' is used as the fusion feature map.

[0180] The fused feature map obtained by adopting this optional implementation method can effectively extract global context information, which can not only better extract signal point features, thereby improving the performance of overall feature extraction, but also achieve feature dimensionality reduction to reduce the amount of calculation and improve the calculation speed.

[0181] It should be noted that the above-mentioned execution entity may first execute step 202 and then execute step 203, or first execute step 203 and then execute step 202, and the present disclosure does not make specific limitations on this.

[0182] Step 204: based on the cell segmentation results of each cell in the cell nucleus image, extract the cell region fusion feature map of each cell in the fusion feature map.

[0183] In this embodiment, the execution subject may adopt various existing or future developed region of interest feature extraction methods, and based on the cell segmentation results of each cell in the cell nucleus image, extract the cell region fusion feature map of each cell in the fusion feature map obtained in step 203. That is, for each cell segmentation result obtained in step 202, according to the region of interest indicated by the cell position information therein, the region of interest feature extraction method is used to locate the region of interest in the fusion feature map and extract features, and obtain the cell region fusion feature map corresponding to the cell segmentation result.

[0184] As an example, the region of interest feature extraction method may include but is not limited to: RoIPooling (Region of Interest Pooling), RoIAlign (Region of Interest Align), and the like.

[0185] Step 205 , based on the cell region mask feature extraction model, feature fusion and extraction are performed on the cell image mask and the cell region fusion feature map of each cell to obtain a masked cell region fusion feature map of the corresponding cell.

[0186] In this embodiment, the executor of the signal point recognition method can perform feature fusion and extraction on the cell image mask and the cell region fusion feature map of each cell based on the cell region mask feature extraction model to obtain the masked cell region fusion feature map of the corresponding cell.

[0187] Here, the cell region mask feature extraction model is used to perform feature fusion on the cell image mask and the cell region fusion feature map of each cell. The fused feature is equivalent to retaining the cell pixel part of the cell region fusion feature map corresponding to the cell image mask, while the background pixel part is specially processed so that it has no feature meaning (for example, set to a specific pixel value). Therefore, the fused feature is equivalent to further refining the cell region fusion feature map and removing the features of the background pixel part. Feature extraction is then performed based on the fused feature to obtain the masked cell region fusion feature map of the corresponding cell.

[0188] In some optional implementations, step 205 may be performed as follows:

[0189] Based on the cell segmentation result of each cell, the cell image mask of the cell is applied to the cell region fusion feature map of the cell to obtain a preliminary masked feature map of the cell, and feature fusion is performed on the preliminary masked feature map of the cell to obtain a masked cell region fusion feature map of the cell.

[0190] Here, for each cell obtained by segmentation in step 202, first obtain the cell image mask in the cell segmentation result of the cell, and then apply the cell image mask of the cell to the cell region fusion feature map of the cell to obtain a preliminary masked feature map of the cell, so that the portion corresponding to the background pixels in the preliminary masked feature map is covered (for example, assigned a value of 0), while the portion corresponding to the cell pixels is retained. Next, further feature fusion is performed on the preliminary masked feature map of the cell to obtain the masked cell region fusion feature map of the cell, so that the cell image mask and the cell region fusion feature map of the cell are better fused. Optionally, the feature fusion of the preliminary masked feature map of the cell here can be achieved through a layer of convolutional neural network, that is, no feature dimensionality reduction is required, only further feature fusion is required.

[0191] For easier understanding, please refer to Figure 5A and Figure 5B .like Figure 5A As shown, the cell image mask 501 acts on the cell region fusion feature map 502 to obtain a preliminary masked feature map 503. After a layer of convolution, the preliminary masked feature map 503 obtains a cell region fusion feature map 504. For further reference Figure 5B ,like Figure 5B As shown, in the preliminary masked feature map 503 obtained after the cell image mask 501 is applied to the cell region fusion feature map 502, the cell pixel part is retained, while the background pixel part is masked as black.

[0192] Step 206, based on the signal point recognition model, signal point recognition is performed on the masked cell region fusion feature map of each cell to obtain the signal point recognition results of the corresponding cell in each channel.

[0193] In this embodiment, the execution subject may input the masked cell region fusion feature map of each cell into the signal point recognition model to obtain the signal point recognition results of the corresponding cell in each channel.

[0194] Here, the signal point recognition model is used to characterize the correspondence between the masked cell area fusion feature map and the signal point recognition results of each channel. In practice, the signal point recognition model can adopt various existing or future developed signal point recognition models, and the present disclosure does not make specific limitations on this.

[0195] In some optional implementations, the signal point recognition result may include a signal point area information map and a signal point heat map of a preset feature map size, and the signal point recognition model may include a signal point area information map extraction model and a signal point heat map extraction model. Accordingly, step 206 may include the following steps: Figure 2E Steps 2061 to 2063 are shown:

[0196] Step 2061, based on the signal point region information map extraction model, perform signal point region information map extraction on the masked cell region fusion feature map of each cell to obtain the signal point region information map of the corresponding cell in each channel.

[0197] Here, the signal point area information map is used to map each pixel point in the feature map under a preset feature map size to an image area in the fluorescent signal point image.

[0198] It is understandable that in order to reduce model parameters and reduce the amount of calculation, feature extraction and dimensionality reduction will be performed on the masked cell area fusion feature map in the signal point heat map extraction model and the signal point area information map extraction model, and the signal point heat map and signal point area information map size (i.e., the preset feature map size) output by the signal point heat map extraction model and the signal point area information map extraction model should be as small as possible, and in most cases will be smaller than the image size of the masked cell area fusion feature map. In addition, if the masked cell area fusion feature map is mapped to the fluorescent signal point image, the area size of the mapped fluorescent signal point image will be larger than the image size of the masked cell area fusion feature map. In other words, if the signal point heat map and the signal point area information map are mapped to the fluorescent signal point image, the area size of the mapped fluorescent signal point image will also be larger than the image size of the signal point heat map and the signal point area information map. In order to locate the signal point in the fluorescent signal point image, it is necessary to map each pixel point in the signal point heat map of a preset feature map size into the fluorescent signal point image through the signal point area information corresponding to the corresponding pixel point in the signal point area information map, and then determine the probability value of the corresponding image area in the mapped fluorescent signal point image being the signal point area.

[0199] Continue to use the above four channel signal point heat maps: HM go 、HM r 、HM b and H.M. gr For example, assuming that the signal point area information graphs corresponding to the above four channels of Gold, Red, Blue and Green are AI go 、AI r 、AI b and AI gr AI information diagram of the above four signal point areas go 、AI r 、AIb and AI gr The image size is the same as the preset feature map size, which can be 64*64. Taking the Gold channel as an example, for the signal point area information map AI go The median coordinate is (x go ,y go ) pixels, assuming that according to (x go ,y go The signal point area information of the pixel is mapped to the area in the Gold channel fluorescence signal point image as (x go ,y go ) is the center point coordinate, the height is H, and the width is W of the area D go , then the signal point area information graph AI go The median coordinate is (x go ,y go ) may include the center point coordinates (x go ,y go ), height H and width W.

[0200] Step 2062: Based on the signal point heat map extraction model, perform signal point heat map extraction on the masked cell region fusion feature map of each cell to obtain the signal point heat map of the corresponding cell in each channel.

[0201] Here, the signal point heat map includes the probability that each pixel point in the feature map at a preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map.

[0202] That is, the signal point heat map of the corresponding cell in each channel can be obtained by inputting the masked cell area fusion feature map of each cell into the signal point heat map extraction model.

[0203] For example, when four channels, Gold, Red, Blue and Green, are included, four signal point heat maps corresponding to the above four channels can be obtained: HM go 、HM r 、HM b and H.M. gr . And the above four signal point heat maps HM go 、HM r 、HM b and H.M. gr The image sizes of the four signal point heat maps HM are the same, which are all preset feature map sizes, for example, 64*64. go 、HM r 、HM b and H.M. grThe pixel value of each pixel in can be a probability value between 0 and 1. go The pixel value of each pixel in is used to characterize the probability that the pixel is a signal point area after being mapped to the image area in the fluorescent signal point image according to the area information map of the corresponding channel (i.e., the Gold channel). In the above example, the coordinates in the signal point area information map of the Gold channel are (x go ,y go ) The pixel point can be mapped to the area D in the Gold channel fluorescence signal point image after the signal point area information is mapped. go , assuming that the signal point heat map HM go The median coordinate is (x go ,y go ) is the probability value corresponding to the pixel point go , then the area D in the Gold channel fluorescence signal point image go The probability of being a signal point area is P go .

[0204] Step 2063, based on the signal point heat map and signal point area information map of each cell in each channel, determine the signal point recognition result of the corresponding cell in each channel.

[0205] Here, the fluorescence signal point image area of ​​the corresponding channel can be located through the signal point area information map of each cell in each channel, and then the probability that the corresponding fluorescence signal point image area is the signal point area can be determined based on the signal point heat map of each channel.

[0206] Finally, the signal point recognition result of each cell in each channel is determined according to the probability that the fluorescent signal point image area is the signal point area.

[0207] Specifically, for each pixel in the fluorescent signal point image area, the probability that the pixel is a signal point area can be determined as the probability that the fluorescent signal point image area to which the pixel belongs is a signal point area, and then the probability that each pixel in the fluorescent signal point image is a signal point pixel can be obtained.

[0208] Then, the probability that each pixel in the fluorescent signal point image is a signal point pixel can be used as the signal point recognition result.

[0209] Alternatively, a signal point mask image can be generated based on the comparison result of the probability that each pixel in the fluorescence signal point image is a signal point pixel and a preset signal point probability threshold to characterize whether each pixel in the fluorescence signal point image of each channel is a signal point pixel, and the above signal point mask image can be used as the signal point recognition result.

[0210] Alternatively, for each pixel in the signal point heat map, the pixel whose corresponding probability value is greater than the preset signal point threshold is determined as the signal point heat map pixel. Then, the pixel whose probability value corresponding to the pixel in the signal point heat map in each channel is greater than the preset signal point threshold is determined as the signal point area pixel, and then the signal point area pixel is mapped to the signal point image area in the fluorescent signal point image, and finally the area information of each mapped signal point image area is determined as the signal point recognition result. For example, the signal point recognition result may include the key point position (the key point may be, for example, the center point or any vertex coordinate of a rectangle) of the signal point image area in the fluorescent signal point image of each channel and the area size information (for example, the height and width information of the rectangle corresponding to the signal point image area).

[0211] In some optional embodiments, the signal point heat map extraction model may include a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0212] Here, the first image feature extraction network and the second image feature extraction network are used to gradually extract image features. The subsequent convolutional layer is used to further extract features and feature dimensionality reduction, and the final activation function layer is used to map the input data to a probability value between 0 and 1. For example, the activation function layer here can be a Sigmoid activation function.

[0213] As an example, both the first image feature extraction network and the second image feature extraction network may be CBL networks. The CBL network includes a convolution layer, a batch normalization layer, and an activation function layer (Leaky ReLU Layer) connected in sequence.

[0214] In some optional implementations, the signal point area information graph extraction model may include a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0215] The signal point width extraction model is used to characterize the correspondence between the masked cell area fusion feature map and the signal point area width map of each channel. The image size of the signal point area width map is the preset feature size (for example, 64*64), and the pixel value corresponding to each pixel in the signal point area width map is the width of the image area mapped to the fluorescence signal point image by the pixel.

[0216] The signal point height extraction model is used to characterize the correspondence between the masked cell area fusion feature map and the signal point area height map of each channel. The image size of the signal point area height map is the preset feature size (for example, 64*64), and the pixel value corresponding to each pixel in the signal point area height map is the height of the image area mapped to the fluorescence signal point image by the pixel.

[0217] The signal point position horizontal coordinate offset extraction model is used to characterize the correspondence between the masked cell area fusion feature map and the signal point position horizontal coordinate offset map of each channel. The image size of the signal point position horizontal coordinate offset height map is a preset feature size (e.g., 64*64), and the pixel value corresponding to each pixel point in the signal point position horizontal coordinate offset map is used to map the pixel point coordinate to the key point horizontal coordinate of the image area in the fluorescent signal point image.

[0218] The signal point position ordinate offset extraction model is used to characterize the correspondence between the masked cell region fusion feature map and the signal point position ordinate offset map of each channel. The image size of the signal point position ordinate offset height map is a preset feature size (e.g., 64*64), and the pixel value corresponding to each pixel in the signal point position ordinate offset map is used to map the pixel coordinates to the key point ordinate of the image area in the fluorescent signal point image.

[0219] Taking the Gold channel as an example, for the signal point area information graph AI go The median coordinate is (x go ,y go ) pixel point, the coordinates in the signal point area width map are (x go ,y go ) corresponds to a pixel value W, and the coordinates in the signal point area height map are (x go ,y go ) corresponds to a pixel value of H, and the coordinates of the signal point position in the horizontal coordinate offset diagram are (x go ,y go ) corresponds to the pixel value Offset_W, and the coordinates of the signal point position ordinate offset in the figure are (x go ,y go ) corresponding to the pixel value Offset_H, indicating that the pixel with the coordinate (x go ,y go ) is mapped to the area of ​​the Gold channel fluorescence signal point image as (x go+ Offset_W,y go+ Offset_H) is the key point coordinate, the height is H, the width is W of the area D go .

[0220] In some optional embodiments, the signal point width extraction model may include a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model may include a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model may include a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model may include a sixth image feature extraction network and a convolutional layer connected in sequence.

[0221] Among them, the third image feature extraction network, the fourth image feature extraction network, the fifth image feature extraction network and the sixth image feature extraction network can all be the above-mentioned CBL network.

[0222] In some optional embodiments, the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model can be obtained by Fig. 6A The training step 600 shown is pre-trained to obtain:

[0223] Step 601 : Based on the sample cell nuclear image and the labeled cell segmentation result of each cell in the sample cell nuclear image, a cell instance segmentation model is trained.

[0224] Here, the labeled cell segmentation result may include a labeled cell image mask and labeled cell position information.

[0225] The labeled cell image mask is used to characterize whether the pixel point at the corresponding position of the cell image is a cell pixel point or a background pixel point. The labeled cell position information is used to characterize the position or area (e.g., a rectangular area) of the cell in the cell nucleus image. As an example, the labeled cell position information may include the coordinates of the four vertices of the rectangular area where the cell is located, or the coordinates of a vertex and the width and height of the rectangular area.

[0226] It can be understood that the data format of the labeled cell image mask and the labeled cell position information here can be the same as the data format of the cell image mask and the cell position information output by the cell instance segmentation model.

[0227] Specifically, first, a large number of sample cell nucleus images can be collected as a sample cell nucleus image dataset. Then, each sample cell nucleus image is annotated to mark the location of the cell nucleus region and the cell image mask. Thus, the sample cell nucleus image can be input into the cell instance segmentation model to obtain the actual cell segmentation result, and then the model parameters of the cell instance segmentation model can be adjusted according to the difference between the actual cell segmentation result and the annotated cell segmentation result.

[0228] Through step 601, a separate training of a cell instance segmentation model can be achieved.

[0229] Step 602, based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, as well as the labeled cell segmentation results of each cell in the sample cell nuclear image and the labeled signal point recognition results in each channel, a fusion feature extraction model, a cell area mask feature extraction model and a signal point recognition model are trained.

[0230] Specifically, here, at the same time as the sample cell nuclear image data set is collected in step 601, the fluorescence signal point images of each channel corresponding to each sample cell nuclear image can be collected. Then, for each cell segmented in the annotated cell segmentation result corresponding to the sample cell nuclear image, the signal point recognition result is annotated in the image area corresponding to the cell in the fluorescence signal point image of each channel. Finally, based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, the annotated cell segmentation result of each cell in the sample cell nuclear image and the annotated signal point recognition result in each channel, a fusion feature extraction model, a cell area mask feature extraction model and a signal point recognition model can be trained.

[0231] That is, in step 602, the cell segmentation result actually output by the cell instance segmentation model is not used, but the labeled cell segmentation result is used. That is to say, in step 602, the labeled cell segmentation result and the cell category label are used to separately train the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model, so as to avoid the segmentation effect of the cell instance segmentation model affecting the training effect of the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0232] Optionally, step 602 may include: Figure 6B The following steps 6021 to 6025 are shown:

[0233] Step 6021: Based on the initial fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a first sample fusion feature map.

[0234] Here, the initial fusion feature extraction model is used to extract image features from the fluorescence signal point images of each channel corresponding to the sample cell nucleus image, and then fuse the image features extracted from the fluorescence signal point images of each channel to obtain the first sample fusion feature map. Thus, the image features of the fluorescence signal point images of each channel corresponding to the sample cell nucleus image are learned in the first sample fusion feature map.

[0235] Specifically, the specific implementation of step 6021 and the model structure of the initial fusion feature extraction model can be referred to in reference Figure 2A The implementation method of step 203 and the related description of the fusion feature extraction model in the corresponding embodiment will not be repeated here.

[0236] Step 6022: extracting the first sample cell region fusion feature map of each cell in the first sample fusion feature map based on the labeled cell position information of each cell in the sample cell nuclear image.

[0237] Specifically, you can use the reference Figure 2A The same region of interest feature extraction method in step 204 in the corresponding embodiment, based on the labeled cell position information of each cell in the sample cell nucleus image, extracts the first sample cell region fusion feature map of each cell in the first sample fusion feature map, which will not be repeated here.

[0238] Since the labeled cell position information of each cell in the sample cell nucleus image accurately provides the cell location and area, the position of the first sample cell area fusion feature map of each cell extracted in the sample fusion feature map according to the labeled cell position information is accurate. This can avoid the possible impact of the segmentation effect of the cell instance segmentation model on other subsequent steps, and can avoid affecting the training process of the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0239] Step 6023, based on the initial cell region mask feature extraction model, feature fusion and extraction are performed on the labeled cell image mask of each cell in the sample cell nucleus image and the first sample cell region fusion feature map to obtain the first sample masked cell region fusion feature map of the corresponding cell.

[0240] Here, the initial cell region mask feature extraction model is used to perform feature fusion on the labeled cell image mask of each cell in the sample cell nucleus image and the first sample cell region fusion feature map. The fused feature is equivalent to retaining the cell pixel points in the labeled cell image mask corresponding to the first sample cell region fusion feature map, while the background pixel points are specially processed so that they do not have feature meaning (for example, set to a specific pixel value). The fused feature is equivalent to further refining the first sample cell region fusion feature map according to the correct labeled labeled cell image mask, and removing the features of the background pixel points. Feature extraction is then performed based on the fused feature to obtain the first sample masked cell region fusion feature map of the corresponding cell.

[0241] It should be noted that, here, feature fusion is performed using the labeled cell image mask and the first sample cell area fusion feature map, which can avoid the possible influence of the segmentation effect of the cell instance segmentation model on other subsequent steps, and can avoid the influence on the training effects of the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0242] Specifically, the specific implementation of step 6023 and the model structure of the initial cell region mask feature extraction model can be referred to in reference Figure 2A The implementation method of step 205 and the related description of the cell region mask feature extraction model in the corresponding embodiment will not be repeated here.

[0243] It should be noted that, after step 6023, the features of the annotated background pixel points are removed from the cell region fusion feature map after the first sample masking, and only the features of the cell pixel points are retained.

[0244] Step 6024, based on the initial signal point recognition model, signal point recognition is performed on the first sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain a first actual signal point recognition result of the corresponding cell.

[0245] Specifically, the specific implementation of step 6024 and the model structure of the initial signal point recognition model can be referred to in reference Figure 2A The implementation method of step 206 and the related description of the signal point recognition model in the corresponding embodiment will not be repeated here.

[0246] Step 6025, based on the difference between the first actual signal point recognition result and the labeled signal point recognition result of each cell in the obtained sample cell nucleus image, adjust the model parameters of the initial fusion feature extraction model, the initial cell area mask feature extraction model and the initial signal point recognition model to obtain the trained fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0247] Here, various loss function calculation methods can be used to calculate the difference between the first actual signal point recognition result and the labeled signal point recognition result of each cell in the obtained sample cell nucleus image, for example, Mean Squared Error (MSE), Mean Absolute Error (MAE), Cross-Entropy Loss, Logarithmic Loss, Huber Loss, etc.

[0248] Various parameter adjustment methods may also be used to adjust the model parameters of the initial fusion feature extraction model, the initial cell region mask feature extraction model, and the initial signal point recognition model. For example, including but not limited to Gradient Descent, Stochastic Gradient Descent (SGD), Mini-batch Gradient Descent, Nesterov Accelerated Gradient (NAG), AdaGrad, RMSProp, Adam (Adaptive Moment Estimation), AdaDelta, Nadam (Nesterov-accelerated Adaptive Moment Estimation), Learning Rate Decay, Early Stopping, Weight Decay, Dropout, BatchNormalization, or a combination of at least two of them.

[0249] After step 602, it is possible to separately train the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model without introducing the cell instance segmentation model, thereby avoiding the segmentation effect of the cell instance segmentation model obtained by training in step 601 alone affecting the training effect of the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model.

[0250] In some optional implementations, the training step 600 may further include the following step 603 after step 602:

[0251] Step 603, based on the sample cell nuclear image, the annotated cell segmentation results of each cell in the sample cell nuclear image and the annotated signal point recognition results in each channel, and the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, a cell instance segmentation model, a fusion feature extraction model, a cell region mask feature extraction model and a signal point recognition model are trained.

[0252] That is to say, after the cell instance segmentation model is trained separately in step 601, and the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are trained separately in step 602, the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are jointly trained by using the sample cell nucleus image, the annotated cell segmentation results of each cell in the sample cell nucleus image and the annotated signal point recognition results in each channel and the fluorescence signal point images of each channel corresponding to the sample cell nucleus image, so as to further improve the training effects of the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model.

[0253] Specifically, step 603 may include: Figure 6C Steps 6031 to 6036 shown:

[0254] Step 6031, based on the cell instance segmentation model, instance segmentation is performed on the sample cell nucleus image to obtain actual cell segmentation results of each cell in the sample cell nucleus image.

[0255] Here, the cell instance segmentation model has been trained in step 601 and has achieved a preliminary segmentation effect. The actual cell segmentation result obtained by performing instance segmentation on the sample cell nucleus image using the cell instance segmentation model can reflect the current segmentation effect of the cell instance segmentation model.

[0256] The actual cell segmentation result may include an actual cell image mask and actual cell position information. For more information about the actual cell image mask and actual cell position information, please refer to Figure 2A The description related to the cell image mask and the cell position information in step 201 in the corresponding embodiment will not be repeated here.

[0257] Step 6032: Based on the fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a second sample fusion feature map.

[0258] Here, the specific operation steps of step 6032 and the technical effects brought about by it can be referred to Figure 2A For the relevant description of step 203 in the corresponding embodiment, please refer to FIG. 6 , which will not be described again here.

[0259] It should be noted that the fusion feature extraction model here has achieved the effect of preliminary feature fusion after training in step 602. The second sample fusion feature map obtained by using the fusion feature extraction model for feature fusion incorporates the signal point features of the fluorescence signal point images of each channel corresponding to the sample cell nucleus image.

[0260] Step 6033, based on the actual cell position information of each cell in the sample cell nucleus image, extract the second sample cell area fusion feature map of each cell in the second sample fusion feature map.

[0261] Here, the specific operation steps of step 6033 and the technical effects brought about by it can be referred to Figure 2A The relevant description of step 204 in the corresponding embodiment will not be repeated here.

[0262] Step 6034, based on the cell region mask feature extraction model, feature fusion and extraction are performed on the actual cell image mask of each cell in the sample cell nucleus image and the second sample cell region fusion feature map to obtain the second sample masked cell region fusion feature map of the corresponding cell.

[0263] Here, the specific operation steps of step 6034 and the technical effects brought about by them can be referred to Figure 2A The relevant description of step 205 in the corresponding embodiment will not be repeated here.

[0264] It should be noted that the cell region mask feature extraction model here has the effect of mask feature fusion and feature extraction after training in step 602. The cell region mask feature extraction model is used to perform feature fusion on the actual cell image mask of each cell in the sample cell nucleus image and the second sample cell region fusion feature map. The fused feature is equivalent to retaining the part of the cell pixels in the actual cell image mask corresponding to the second sample cell region fusion feature map, while the background pixel part is specially processed so that it has no feature meaning (for example, set to a specific pixel value). Then, the fusion feature map is equivalent to further refining the actual cell image mask output by the cell instance segmentation model in step 6031 for the second sample cell region fusion feature map, and removing the features of the background pixel part. Feature extraction is then performed based on the fused features to obtain the second sample masked cell region fusion feature map of the corresponding cell.

[0265] Step 6035: Based on the signal point recognition model, signal point recognition is performed on the second sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain a second actual signal point recognition result of the corresponding cell.

[0266] Here, the specific operation steps of step 6035 and the technical effects brought about by it can be referred to Figure 2A The relevant description of step 206 in the corresponding embodiment will not be repeated here.

[0267] Step 6036, based on the difference between the second actual cell segmentation result and the labeled cell segmentation result of each cell in the obtained sample cell nucleus image and the difference between the second actual signal point recognition result and the labeled signal point recognition result, adjust the model parameters of the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model to obtain the trained cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0268] It should be noted that the difference between step 6036 and step 6025 is that: first, when calculating the loss function, in addition to calculating the difference between the second actual signal point recognition result of the cell and the marked signal point recognition result, the difference between the second actual cell segmentation result of the cell and the marked cell segmentation result is also calculated. That is, the situation that the model of the cell instance segmentation model may have segmentation errors is also considered here. Second, when adjusting the model parameters, in addition to adjusting the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model, the model parameters of the cell instance segmentation model are also adjusted synchronously. That is, the joint training of the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model is realized to further improve the model effects of the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model.

[0269] For the specific treatment of the loss function calculation method and the model parameter optimization method, the technical effects brought by it can be referred to Figure 6B The relevant description of step 6025 in the corresponding embodiment will not be repeated here.

[0270] It should be noted that the above-mentioned various optional implementation modes can be combined arbitrarily according to actual needs.

[0271] The signal point recognition method provided by the above-mentioned embodiment of the present disclosure first performs instance segmentation on the cell nucleus image based on the cell instance segmentation model to obtain the cell segmentation results of each cell in the cell nucleus image; then, based on the fusion feature extraction model, image feature extraction is performed on the fluorescence signal point image of each channel to obtain a fusion feature map; then, based on the cell segmentation results of each cell, the cell region fusion feature map of each cell is extracted in the fusion feature map; then, based on the cell region mask feature extraction model, feature fusion and extraction are performed on the cell image mask and the cell region fusion feature map of each cell to obtain the masked cell region fusion feature map of the corresponding cell; finally, based on the signal point recognition model, signal point recognition is performed on the masked cell region fusion feature map of each cell to obtain the signal point recognition results of the corresponding cell in each channel. Thus, by inputting the cell nucleus image and the fluorescence signal point images of each channel taken under the same field of view, the cell position of each cell in the cell nucleus image, the cell image mask and the signal point recognition results of the corresponding cell in each channel can be output. This enables end-to-end cell segmentation and signal point recognition based on cell nucleus images and multi-channel fluorescence signal point images, and can simultaneously complete cell segmentation and signal point recognition with fewer model parameters, saving computational effort and improving computational efficiency.

[0272] Reference below Figure 7 , Figure 7 FIG. 1 is a schematic diagram of an application scenario of an embodiment of the signal point recognition method disclosed in the present invention. Figure 7 As shown, in Figure 7In the application scenario, the cell nucleus image 701A (i.e., Dapi image) is input into the cell instance segmentation model 702 to obtain the cell segmentation result 703, and the cell segmentation result includes the cell image mask 703A; the fluorescent signal point images 701B, 701C, 701D and 701E of the four channels (i.e., Red channel, Green channel, Gold channel and Aqua channel) corresponding to the cell nucleus image 701A are sequentially input into the primary fusion feature extraction model 704A and the multi-scale feature extraction model 704B to obtain the fusion feature map 705; the primary fusion feature extraction model 704A and the multi-scale feature extraction model 704B together constitute the fusion feature extraction model 704; based on the cell segmentation result 703, the cell instance segmentation result 703 of each cell is extracted in the fusion feature map 705. cell area fusion feature map 706; based on the cell area mask feature extraction model 707, the cell image mask 703A and the cell area fusion feature map 706 of each cell are subjected to feature fusion and extraction to obtain the masked cell area fusion feature map of the corresponding cell; the masked cell area fusion feature map of each cell is respectively input into the signal point heat map extraction model 708A, the signal point width extraction model 708B, the signal point height extraction model 708C, the signal point position horizontal coordinate offset extraction model 708D and the signal point position vertical coordinate offset extraction model 708E to obtain the signal point heat map 709A, the signal point width map 709B, the signal point height map 709C, the signal point position horizontal coordinate offset map 709D and the signal point position vertical coordinate offset map 709E of the corresponding cell in each channel.

[0273] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a signal point identification device. Figure 2A Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0274] like Figure 8 As shown, the signal point recognition device 800 of this embodiment includes: an acquisition module 801, a cell segmentation module 802, a signal point feature fusion module 803, a regional signal point feature extraction module 804, a mask processing module 805 and a signal point recognition module 806.

[0275] The acquisition module 801 is configured to acquire the cell nucleus image and the fluorescence signal point image of each channel captured in the same field of view;

[0276] The cell segmentation module 802 is configured to perform instance segmentation on the cell nucleus image based on a cell instance segmentation model to obtain a cell segmentation result of each cell in the cell nucleus image, wherein the cell segmentation result includes a cell image mask and cell position information;

[0277] The signal point feature fusion module 803 is configured to extract image features from the fluorescence signal point images of each channel based on the fusion feature extraction model to obtain a fusion feature map;

[0278] The regional signal point feature extraction module 804 is configured to extract a cell region fusion feature map of each cell in the fusion feature map based on the cell segmentation results of each cell in the cell nucleus image;

[0279] The mask processing module 805 is configured to perform feature fusion and extraction on the cell image mask and the cell region fusion feature map of each cell based on the cell region mask feature extraction model to obtain the masked cell region fusion feature map of the corresponding cell;

[0280] The signal point recognition module 806 is configured to perform signal point recognition on the masked cell region fusion feature map of each cell based on the signal point recognition model to obtain the signal point recognition results of the corresponding cell in each channel.

[0281] In this embodiment, the specific processing of the acquisition module 801, the cell segmentation module 802, the signal point feature fusion module 803, the regional signal point feature extraction module 804, the mask processing module 805 and the signal point recognition module 806 of the signal point recognition device 800 and the technical effects thereof can be referred to respectively. Figure 2A The relevant descriptions of step 201, step 202, step 203, step 204, step 205 and step 206 in the corresponding embodiment are not repeated here.

[0282] In some optional implementations, the fusion feature extraction model may include a primary fusion feature extraction model and a multi-scale feature extraction model; and

[0283] The signal point feature fusion module 803 may include:

[0284] The feature fusion unit 8031 ​​is configured to extract image features from the fluorescence signal point images of each channel based on the primary fusion feature extraction model to obtain a preliminary fusion feature map;

[0285] The multi-scale feature extraction unit 8032 is configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain the fusion feature map.

[0286] In some optional implementations, the feature fusion unit 8031 ​​may be further configured as follows:

[0287] Based on multiple continuous depth-wise separable convolutional layers, image feature extraction is performed on the fluorescence signal point image of each channel in turn, and channel image features of each channel output by the last depth-wise separable convolutional layer are obtained, wherein the convolution kernels in each depth-wise separable convolutional layer are divided into multiple groups, the number of groups of convolution kernels is the same as the number of channels corresponding to each of the fluorescence signal point images, each group of convolution kernels is used to extract image features from the fluorescence signal point image of the corresponding channel, and the number of convolution kernels of the multiple continuous depth-wise separable convolutional layers increases in turn with the increase of convolutional layers;

[0288] Feature fusion is performed on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map.

[0289] In some optional implementations, the step of performing feature fusion on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map may include:

[0290] The concatenation result of the channel image features of each channel is convolved to obtain the preliminary fusion feature map.

[0291] In some optional embodiments, the multi-scale feature extraction model may include a multi-scale feature extraction network and a feature pyramid network; and

[0292] The multi-scale feature extraction unit 8032 may be further configured as follows:

[0293] Based on a multi-scale feature extraction network, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale down-sampling feature map;

[0294] Inputting the multi-scale downsampled feature maps into a feature pyramid network from top to bottom in order of scale from small to large;

[0295] The output result of the bottom layer of the feature pyramid network is determined as the fused feature map.

[0296] In some optional implementations, the mask processing module 805 may be further configured to:

[0297] Based on the cell segmentation result of each cell, the cell image mask of the cell is applied to the cell region fusion feature map of the cell to obtain a preliminary masked feature map of the cell, and feature fusion is performed on the preliminary masked feature map of the cell to obtain a masked cell region fusion feature map of the cell.

[0298] In some optional implementations, the signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and

[0299] The signal point identification module 806 may be further configured to:

[0300] Based on the signal point area information map extraction model, the signal point area information map is extracted from the masked cell area fusion feature map of each cell to obtain the signal point area information map of the corresponding cell in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescent signal point image;

[0301] Based on the signal point heat map extraction model, a signal point heat map is extracted from the masked cell area fusion feature map of each cell to obtain a signal point heat map of the corresponding cell in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map;

[0302] Based on the signal point area information map and signal point heat map of each cell in each channel, the signal point identification results of the corresponding cells in each channel are determined.

[0303] In some optional embodiments, the signal point heat map extraction model may include a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0304] In some optional embodiments, the first image feature extraction network and the second image feature extraction network may respectively include a convolution layer, a batch normalization layer, and an activation function layer connected in sequence.

[0305] In some optional implementations, the signal point area information graph extraction model may include a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0306] In some optional embodiments, the signal point width extraction model may include a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model may include a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model may include a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model may include a sixth image feature extraction network and a convolutional layer connected in sequence.

[0307] In some optional embodiments, the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model can be pre-trained through the following training steps:

[0308] Based on the sample cell nuclear image and the labeled cell segmentation result of each cell in the sample cell nuclear image, the cell instance segmentation model is trained, and the labeled cell segmentation result includes a labeled cell image mask and labeled cell position information;

[0309] Based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, as well as the labeled cell segmentation results of each cell in the sample cell nuclear image and the labeled signal point recognition results in each channel, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained to obtain.

[0310] In some optional embodiments, the training of the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, the annotated cell segmentation result of each cell in the sample cell nuclear image and the annotated signal point recognition result in each channel may include:

[0311] Based on the initial fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a first sample fusion feature map;

[0312] Extracting a first sample cell region fusion feature map of each cell in the first sample fusion feature map based on the labeled cell position information of each cell in the sample cell nuclear image;

[0313] Based on the initial cell region mask feature extraction model, feature fusion and extraction are performed on the labeled cell image mask of each cell in the sample cell nucleus image and the first sample cell region fusion feature map to obtain the first sample masked cell region fusion feature map of the corresponding cell;

[0314] Based on the initial signal point recognition model, signal point recognition is performed on the first sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain the first actual signal point recognition result of the corresponding cell in each channel;

[0315] Based on the difference between the first actual signal point recognition result and the labeled signal point recognition result of each cell in the sample cell nucleus image in each channel, the model parameters of the initial fusion feature extraction model, the initial cell area mask feature extraction model and the initial signal point recognition model are adjusted to obtain the trained fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0316] In some optional implementations, the training step may further include:

[0317] Based on the sample cell nuclear image, the labeled cell segmentation result of each cell in the sample cell nuclear image and the labeled signal point recognition result in each channel, and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image, the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained.

[0318] In some optional embodiments, the training of the cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model based on the sample cell nuclear image, the labeled cell segmentation result of each cell in the sample cell nuclear image and the labeled signal point recognition result in each channel and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image may include:

[0319] Based on the cell instance segmentation model, the sample cell nuclear image is instance segmented to obtain actual cell segmentation results of each cell in the sample cell nuclear image, wherein the actual cell segmentation results include actual cell image masks and actual cell position information;

[0320] Based on the fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a second sample fusion feature map;

[0321] Extracting a second sample cell region fusion feature map of each cell in the second sample fusion feature map based on actual cell position information of each cell in the sample cell nucleus image;

[0322] Based on the cell region mask feature extraction model, feature fusion and extraction are performed on the actual cell image mask of each cell in the sample cell nucleus image and the second sample cell region fusion feature map to obtain the second sample masked cell region fusion feature map of the corresponding cell;

[0323] Based on the signal point recognition model, signal point recognition is performed on the second sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain a second actual signal point recognition result of the corresponding cell in each channel;

[0324] Based on the difference between the second actual cell segmentation result and the labeled cell segmentation result of each cell in the sample cell nucleus image, and the difference between the second actual signal point recognition result and the labeled signal point recognition result in each channel, the model parameters of the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are adjusted to obtain the trained cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

[0325] It should be noted that the implementation details and technical effects of each module in the signal point identification device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0326] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 suitable for implementing the electronic device of the present disclosure. Fig. 9 The computer system 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0327] like Fig. 9 As shown, the computer system 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the computer system 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0328] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication devices 909 may allow the computer system 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The computer system 900 of an electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0329] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0330] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0331] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0332] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2A The illustrated embodiment and its optional implementations illustrate a signal point identification method.

[0333] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Python, Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0334] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0335] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the acquisition module may also be described as "a module for acquiring cell nucleus images and fluorescence signal point images of each channel taken in the same field of view".

[0336] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A signal point identification method, comprising: Acquire the cell nucleus image and the fluorescence signal point image of each channel taken in the same field of view; Based on the cell instance segmentation model, the cell nucleus image is instance segmented to obtain the cell segmentation result of each cell in the cell nucleus image, wherein the cell segmentation result includes a cell image mask and cell position information; Based on the fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel to obtain a fusion feature map; Based on the cell segmentation results of each cell in the cell nucleus image, extracting a cell region fusion feature map of each cell in the fusion feature map; Based on the cell region mask feature extraction model, the cell image mask and the cell region fusion feature map of each cell are fused and extracted to obtain the masked cell region fusion feature map of the corresponding cell; Based on the signal point recognition model, signal point recognition is performed on the masked cell area fusion feature map of each cell to obtain the signal point recognition results of the corresponding cells in each channel.

2. The method according to claim 1, wherein: The fusion feature extraction model includes a primary fusion feature extraction model and a multi-scale feature extraction model; as well as The method of extracting image features of the fluorescence signal point images of each channel based on the fusion feature extraction model to obtain a fusion feature map includes: Based on the primary fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel to obtain a preliminary fusion feature map; Based on a multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain the fusion feature map.

3. The method according to claim 2, wherein: Based on the primary fusion feature extraction model, image features are extracted from the fluorescence signal point images of each channel to obtain a preliminary fusion feature map, including: Based on multiple continuous depth-wise separable convolutional layers, image feature extraction is performed on the fluorescence signal point image of each channel in turn, and channel image features of each channel output by the last depth-wise separable convolutional layer are obtained, wherein the convolution kernels in each depth-wise separable convolutional layer are divided into multiple groups, the number of groups of convolution kernels is the same as the number of channels corresponding to each of the fluorescence signal point images, each group of convolution kernels is used to extract image features from the fluorescence signal point image of the corresponding channel, and the number of convolution kernels of the multiple continuous depth-wise separable convolutional layers increases in turn with the increase of convolutional layers; Feature fusion is performed on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map.

4. The method according to claim 3, wherein: The step of performing feature fusion on the splicing results of the channel image features of each channel to obtain the preliminary fusion feature map includes: The concatenation result of the channel image features of each channel is convolved to obtain the preliminary fusion feature map.

5. The method according to claim 2, wherein: The multi-scale feature extraction model includes a multi-scale feature extraction network and a feature pyramid network; as well as The step of performing multi-scale feature extraction on the preliminary fusion feature map based on the multi-scale feature extraction model to obtain the fusion feature map includes: Based on a multi-scale feature extraction network, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale down-sampling feature map; Inputting the multi-scale downsampled feature maps into a feature pyramid network from top to bottom in order of scale from small to large; The output result of the bottom layer of the feature pyramid network is determined as the fused feature map.

6. The method according to claim 1, wherein: The cell region mask feature extraction model is based on which the cell image mask and the cell region fusion feature map of each cell are subjected to feature fusion and extraction to obtain the masked cell region fusion feature map of the corresponding cell, including: Based on the cell segmentation result of each cell, the cell image mask of the cell is applied to the cell region fusion feature map of the cell to obtain a preliminary masked feature map of the cell, and feature fusion is performed on the preliminary masked feature map of the cell to obtain a masked cell region fusion feature map of the cell.

7. The method according to claim 1, wherein: The signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and Based on the signal point recognition model, the signal point recognition is performed on the masked cell area fusion feature map of each cell to obtain the signal point recognition results of the corresponding cell in each channel, including: Based on the signal point area information map extraction model, the signal point area information map is extracted from the masked cell area fusion feature map of each cell to obtain the signal point area information map of the corresponding cell in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescent signal point image; Based on the signal point heat map extraction model, a signal point heat map is extracted from the masked cell area fusion feature map of each cell to obtain a signal point heat map of the corresponding cell in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map; Based on the signal point area information map and signal point heat map of each cell in each channel, the signal point identification results of the corresponding cells in each channel are determined.

8. The method according to claim 7, wherein: The signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer and an activation function layer which are connected in sequence.

9. The method according to claim 8, wherein: The first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

10. The method according to claim 7, wherein: The signal point area information graph extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model and a signal point position vertical coordinate offset extraction model.

11. The method according to claim 10, wherein: The signal point width extraction model includes a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model includes a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model includes a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model includes a sixth image feature extraction network and a convolutional layer connected in sequence.

12. The method according to claim 1, wherein: The cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are pre-trained through the following training steps: Based on the sample cell nuclear image and the labeled cell segmentation result of each cell in the sample cell nuclear image, the cell instance segmentation model is trained, and the labeled cell segmentation result includes a labeled cell image mask and labeled cell position information; Based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, as well as the labeled cell segmentation results of each cell in the sample cell nuclear image and the labeled signal point recognition results in each channel, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained to obtain.

13. The method according to claim 12, wherein: The training of the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model based on the fluorescence signal point images of each channel corresponding to the sample cell nuclear image, the labeled cell segmentation result of each cell in the sample cell nuclear image and the labeled signal point recognition result in each channel includes: Based on the initial fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a first sample fusion feature map; Extracting a first sample cell region fusion feature map of each cell in the first sample fusion feature map based on the labeled cell position information of each cell in the sample cell nuclear image; Based on the initial cell region mask feature extraction model, feature fusion and extraction are performed on the labeled cell image mask of each cell in the sample cell nucleus image and the first sample cell region fusion feature map to obtain the first sample masked cell region fusion feature map of the corresponding cell; Based on the initial signal point recognition model, signal point recognition is performed on the first sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain the first actual signal point recognition result of the corresponding cell in each channel; Based on the difference between the first actual signal point recognition result and the labeled signal point recognition result of each cell in the sample cell nucleus image in each channel, the model parameters of the initial fusion feature extraction model, the initial cell area mask feature extraction model and the initial signal point recognition model are adjusted to obtain the trained fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

14. The method according to claim 13, wherein: The training step also includes: Based on the sample cell nuclear image, the labeled cell segmentation result of each cell in the sample cell nuclear image and the labeled signal point recognition result in each channel, and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image, the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are trained.

15. The method according to claim 14, wherein: The cell instance segmentation model, the fusion feature extraction model, the cell region mask feature extraction model and the signal point recognition model are trained based on the sample cell nuclear image, the labeled cell segmentation result of each cell in the sample cell nuclear image and the labeled signal point recognition result in each channel, and the fluorescence signal point image of each channel corresponding to the sample cell nuclear image, including: Based on the cell instance segmentation model, the sample cell nuclear image is instance segmented to obtain actual cell segmentation results of each cell in the sample cell nuclear image, wherein the actual cell segmentation results include actual cell image masks and actual cell position information; Based on the fusion feature extraction model, image feature extraction is performed on the fluorescence signal point images of each channel corresponding to the sample cell nucleus image to obtain a second sample fusion feature map; Extracting a second sample cell region fusion feature map of each cell in the second sample fusion feature map based on actual cell position information of each cell in the sample cell nucleus image; Based on the cell region mask feature extraction model, feature fusion and extraction are performed on the actual cell image mask of each cell in the sample cell nucleus image and the second sample cell region fusion feature map to obtain the second sample masked cell region fusion feature map of the corresponding cell; Based on the signal point recognition model, signal point recognition is performed on the second sample masked cell area fusion feature map of each cell in the sample cell nucleus image to obtain a second actual signal point recognition result of the corresponding cell in each channel; Based on the difference between the second actual cell segmentation result and the labeled cell segmentation result of each cell in the sample cell nucleus image, and the difference between the second actual signal point recognition result and the labeled signal point recognition result in each channel, the model parameters of the cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model are adjusted to obtain the trained cell instance segmentation model, the fusion feature extraction model, the cell area mask feature extraction model and the signal point recognition model.

16. A signal point identification device, comprising: An acquisition module is configured to acquire a cell nucleus image and a fluorescence signal point image of each channel captured in the same field of view; A cell segmentation module is configured to perform instance segmentation on the cell nucleus image based on a cell instance segmentation model to obtain a cell segmentation result of each cell in the cell nucleus image, wherein the cell segmentation result includes a cell image mask and cell position information; The signal point feature fusion module is configured to extract image features of the fluorescence signal point images of each channel based on the fusion feature extraction model to obtain a fusion feature map; A regional signal point feature extraction module is configured to extract a cell region fusion feature map of each cell in the fusion feature map based on the cell segmentation results of each cell in the cell nucleus image; The mask processing module is configured to perform feature fusion and extraction on the cell image mask and the cell region fusion feature map of each cell based on the cell region mask feature extraction model to obtain the masked cell region fusion feature map of the corresponding cell; The signal point recognition module is configured to perform signal point recognition on the masked cell area fusion feature map of each cell based on the signal point recognition model to obtain the signal point recognition results of the corresponding cell in each channel.

17. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 15 is implemented.

19. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images