Fluorescent signal dot recognition method and related products

Abstract

The present disclosure provides a fluorescent signal point recognition method and related products. A specific embodiment of the method extracts image features from multi-channel fluorescent signal point images to be recognized based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; then multi-scale feature extraction is performed on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; finally, fluorescent signal point recognition is performed on the multi-scale fusion feature map based on a fluorescent signal point recognition model to obtain the fluorescent signal point recognition result of each channel fluorescent signal point image to be recognized. That is, only the multi-channel fluorescent signal point images taken under the same field of view including the same cell object need to be input, and the fluorescent signal point recognition result of each channel fluorescent signal point image can be output, and the fluorescent signal point recognition is completed with fewer model parameters, saving the calculation amount and improving the calculation efficiency.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of image recognition, and particularly to a fluorescent signal point recognition method and related products. BACKGROUND

[0002] The images (including multi-channel fluorescent signal point images and Dapi images) obtained based on fluorescent in situ hybridization (FISH) technology imaging can specifically detect diseases with high sensitivity. For example, lung cancer circulating genetically abnormal cells (CAC) can be detected and judged one by one by four-channel fluorescent signal point images and Dapi images.

[0003] Currently, the method of automatically recognizing CAC cells mostly needs to segment the cells based on the Dapi image to obtain a cell segmentation result, locate each cell in the four-channel fluorescent signal point images, and cut the fluorescent signal point images of the corresponding channel for each cell, and then recognize the fluorescent signal points in each channel of the cut fluorescent signal point images to obtain the fluorescent signal point recognition result of the corresponding channel. Finally, whether the cell is a CAC cell is determined according to the number of fluorescent signal points in the cell. In this process, an important part is to obtain the fluorescent signal point recognition result of each cell in the fluorescent signal point images of each channel. Currently, the above process is divided into two stages of cell segmentation and fluorescent signal point recognition, but the above two-stage method needs to locate each cell in the four-channel signal point images, and then set a corresponding recognition model for each cell in each channel of the signal point images to recognize the fluorescent signal points, which results in more model parameters and thus more storage space and larger calculation amount, and lower calculation efficiency. SUMMARY

[0004] Embodiments of the present disclosure provide a fluorescent signal point recognition method, device, electronic equipment, storage medium and computer program product.

[0005] In a first aspect, embodiments of the present disclosure provide a fluorescent signal point recognition method, which includes: acquiring multi-channel to-be-recognized fluorescent signal point images obtained by photographing the same field of view including the same cell object; performing image feature extraction on the multi-channel to-be-recognized fluorescent signal point images based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; performing multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; and performing fluorescent signal point recognition on the multi-scale fusion feature map based on a fluorescent signal point recognition model to obtain a fluorescent signal point recognition result of each channel of the to-be-recognized fluorescent signal point image.

[0006] In some optional embodiments, the multi-scale feature extraction model comprises a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an up-sampling network, a concatenation network, and a fourth C2f network connected in sequence, and the output of the second C2f network and the up-sampling network is input to the concatenation network.

[0007] In some optional embodiments, the fluorescent signal point recognition result comprises a signal point region information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model comprises a signal point region information map extraction model and a signal point heat map extraction model; and

[0008] The fluorescent signal point recognition model is used to perform fluorescent signal point recognition on the multi-scale fusion feature map to obtain a fluorescent signal point recognition result of each channel of the to-be-recognized fluorescent signal point image, which comprises:

[0009] The signal point region information map extraction model is used to perform signal point region information map extraction on the multi-scale fusion feature map to obtain a signal point region information map of each channel of the to-be-recognized fluorescent signal point image, wherein the signal point region information map is used to map each pixel point in the feature map under the preset feature map size to an image region in the fluorescent signal point image;

[0010] The signal point heat map extraction model is used to perform signal point heat map extraction on the multi-scale fusion feature map to obtain a signal point heat map of each channel of the to-be-recognized fluorescent signal point image, wherein the signal point heat map comprises a probability of each pixel point in the feature map under the preset feature map size being a signal point region after being mapped to an image region in the fluorescent signal point image according to the signal point region information map;

[0011] For each channel of the to-be-recognized fluorescent signal point image, the fluorescent signal point recognition result of the channel of the to-be-recognized fluorescent signal point image is determined based on the signal point region information map and the signal point heat map of the channel of the to-be-recognized fluorescent signal point image.

[0012] In some optional embodiments, the signal point heat map extraction model comprises a first image feature extraction network, a second image feature extraction network, a convolutional layer, and an activation function layer connected in sequence.

[0013] In some optional embodiments, the first image feature extraction network and the second image feature extraction network each comprise a convolutional layer, a batch normalization layer, and an activation function layer connected in sequence.

[0014] In some optional embodiments, the signal point region information map extraction model comprises a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0015] In some optional embodiments, the signal point width extraction model comprises a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model comprises a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model comprises a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model comprises a sixth image feature extraction network and a convolutional layer connected in sequence.

[0016] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model, and the fluorescent signal point recognition model are obtained by the following training steps:

[0017] obtaining a sample data set, wherein the sample data comprises multi-channel sample fluorescent signal point images and labeled fluorescent signal point recognition results of each channel sample fluorescent signal point image, and the multi-channel sample fluorescent signal point image is a multi-channel fluorescent signal point image obtained by photographing the same field of view comprising the same cell object;

[0018] training the preliminary fusion feature extraction model, the multi-scale feature extraction model, and the fluorescent signal point recognition model based on the sample data set.

[0019] In a second aspect, embodiments of the present disclosure provide a fluorescent signal point recognition device, which comprises: an obtaining module configured to obtain a multi-channel to-be-recognized fluorescent signal point image obtained by photographing the same field of view comprising the same cell object; a preliminary fusion feature extraction module configured to perform image feature extraction on the multi-channel to-be-recognized fluorescent signal point image based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; a multi-scale feature extraction module configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; and a fluorescent signal point recognition module configured to perform fluorescent signal point recognition on the multi-scale fusion feature map based on a fluorescent signal point recognition model to obtain a fluorescent signal point recognition result of each channel to-be-recognized fluorescent signal point image.

[0020] In some optional embodiments, the multi-scale feature extraction model comprises a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an up-sampling network, a concatenation network, and a fourth C2f network connected in sequence, and the output of the second C2f network and the up-sampling network is input into the concatenation network.

[0021] In some optional embodiments, the fluorescent signal point recognition result comprises a signal point region information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model comprises a signal point region information map extraction model and a signal point heat map extraction model.

[0022] The fluorescent signal point recognition module is further configured to:

[0023] perform signal point region information map extraction on the multi-scale fusion feature map based on the signal point region information map extraction model to obtain a signal point region information map of each channel of the to-be-recognized fluorescent signal point image, wherein the signal point region information map is used to map each pixel point in the feature map under the preset feature map size to an image region in the fluorescent signal point image;

[0024] perform signal point heat map extraction on the multi-scale fusion feature map based on the signal point heat map extraction model to obtain a signal point heat map of each channel of the to-be-recognized fluorescent signal point image, wherein the signal point heat map comprises a probability of each pixel point in the feature map under the preset feature map size being a signal point region after being mapped to an image region in the fluorescent signal point image according to the signal point region information map;

[0025] For each channel of the to-be-recognized fluorescent signal point image, determine a fluorescent signal point recognition result of the channel of the to-be-recognized fluorescent signal point image based on the signal point region information map and the signal point heat map of the channel of the to-be-recognized fluorescent signal point image.

[0026] In some optional embodiments, the signal point heat map extraction model comprises a first image feature extraction network, a second image feature extraction network, a convolutional layer, and an activation function layer connected in sequence.

[0027] In some optional embodiments, the first image feature extraction network and the second image feature extraction network each comprise a convolutional layer, a batch normalization layer, and an activation function layer connected in sequence.

[0028] In some optional embodiments, the signal point region information map extraction model comprises a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0029] In some optional embodiments, the signal point width extraction model comprises a third image feature extraction network and a convolution layer connected in sequence, the signal point height extraction model comprises a fourth image feature extraction network and a convolution layer connected in sequence, the signal point position horizontal coordinate offset extraction model comprises a fifth image feature extraction network and a convolution layer connected in sequence, and the signal point position vertical coordinate offset extraction model comprises a sixth image feature extraction network and a convolution layer connected in sequence.

[0030] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model, and the fluorescent signal point recognition model are obtained by pre-training through the following training steps:

[0031] obtaining a sample data set, wherein the sample data comprises multi-channel sample fluorescent signal point images and labeled fluorescent signal point recognition results of each channel sample fluorescent signal point image, and the multi-channel sample fluorescent signal point image is a multi-channel fluorescent signal point image obtained by photographing the same field of view comprising the same cell object;

[0032] training the preliminary fusion feature extraction model, the multi-scale feature extraction model, and the fluorescent signal point recognition model based on the sample data set.

[0033] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; and a storage device having one or more programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in any implementation manner of the first aspect.

[0034] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner of the first aspect.

[0035] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising computer programs / instructions, which, when executed by a processor, implement the method described in any implementation manner of the first aspect.

[0036] To solve the problem that the current two-stage fluorescence signal point recognition method requires more model parameters and thus has low calculation efficiency, the fluorescence signal point recognition method, device, electronic equipment, storage medium and computer program product provided by the embodiments of the present disclosure are provided. Through the preliminary fusion feature extraction model, image feature extraction is performed on the overall merging of the multi-channel fluorescence signal point image to be recognized for the same cell object to obtain a preliminary fusion feature map. Then, based on the multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale fusion feature map. Finally, based on the fluorescence signal point recognition model, fluorescence signal point recognition is performed on the multi-scale fusion feature map to obtain the fluorescence signal point recognition result of each channel fluorescence signal point image to be recognized. That is, only the multi-channel fluorescence signal point image photographed in the same field of view of one cell object after cell segmentation needs to be input, and the fluorescence signal point recognition result of each channel fluorescence signal point image can be output. Thus, end-to-end fluorescence signal point recognition can be realized, fluorescence signal point recognition can be realized based on multi-channel fluorescence signal point images, and fluorescence signal point recognition can be completed with fewer model parameters, saving the calculation amount and improving the calculation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0037] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments made with reference to the accompanying drawings. The drawings are for purposes of illustrating the specific embodiments only and are not to be construed as limiting the application. In the drawings:

[0038] Figure 1 is an exemplary system architecture diagram to which one embodiment of the present disclosure can be applied;

[0039] Figure 2 is a flowchart of one embodiment of the fluorescence signal point recognition method according to the present disclosure;

[0040] Figure 3 is a model structure diagram of one embodiment of the fluorescence signal point recognition method according to the present disclosure;

[0041] Figure 4 is a decomposition flowchart of one embodiment of step 204 according to the present disclosure;

[0042] Figure 5 is a flowchart of one embodiment of the training step 500 according to the present disclosure;

[0043] Figure 6 is a structure diagram of one embodiment of the fluorescence signal point recognition device according to the present disclosure;

[0044] Figure 7 is a structure diagram of a computer system of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0045] The present disclosure will be further described in details with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein merely serve illustrative purposes and are not intended to limit the present disclosure. In addition, it should be noted that only the parts related to the present disclosure are shown in the drawings for the convenience of description.

[0046] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the drawings and embodiments.

[0047] Figure 1 An exemplary system architecture 100 of the embodiments of the fluorescent signal point recognition method, device, electronic device and storage medium to which the present disclosure can be applied is shown.

[0048] As shown in Figure 1 The system architecture 100 can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0049] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as fluorescent signal point recognition applications, short video social applications, audio and video conference applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0050] The terminal device 101, 102, 103 can be hardware or software. When the terminal device 101, 102, 103 is hardware, it can be various electronic devices with an information input device (for example, a keyboard, a mouse, a touch screen, a microphone, a camera, etc.) and an information output device (for example, a display screen, a loudspeaker, etc.), including but not limited to a smart phone, a tablet computer, an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, a desktop computer, etc. When the terminal device 101, 102, 103 is software, it can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, to provide the fluorescent signal point identification service), or as a single software or software module. No specific limitation is made herein.

[0051] In some cases, the fluorescent signal point identification method provided by the present disclosure can be performed by the terminal device 101, 102, 103, and accordingly, the fluorescent signal point identification apparatus can be arranged in the terminal device 101, 102, 103. At this time, the system architecture 100 can also not include the server 105.

[0052] In some cases, the fluorescent signal point identification method provided by the present disclosure can be performed by the terminal device 101, 102, 103 and the server 105 together, for example, the step of "obtaining a multi-channel to-be-identified fluorescent signal point image obtained by photographing the same field of view including the same cell object" can be performed by the terminal device 101, 102, 103, and the steps of "performing image feature extraction on the multi-channel to-be-identified fluorescent signal point image based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map" and the like can be performed by the server 105. The present disclosure does not make any limitation in this regard. Accordingly, the fluorescent signal point identification apparatus can also be arranged in the terminal device 101, 102, 103 and the server 105, respectively.

[0053] In some cases, the fluorescent signal point identification method provided by the present disclosure can be performed by the server 105, and accordingly, the fluorescent signal point identification apparatus can also be arranged in the server 105, at this time, the system architecture 100 can also not include the terminal device 101, 102, 103.

[0054] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0055] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0056] Continuing to refer to Figure 2 which shows a flow 200 of one embodiment of a fluorescent signal point recognition method according to the present disclosure, which includes the following steps:

[0057] Step 201, acquiring a multi-channel to-be-recognized fluorescent signal point image obtained by photographing the same field of view including the same cell object.

[0058] In the present embodiment, the execution subject (for example, the terminal device shown in Figure 1 The execution subject can acquire the multi-channel to-be-recognized fluorescent signal point image obtained by photographing the same field of view including the same cell object locally or remotely from other electronic devices connected to the network of the execution subject.

[0059] Here, the multi-channel fluorescent signal point image obtained by photographing the same field of view including the same cell object by a microscope (for example, the fluorescent image photographed under four channels of Red, Green, Gold, and Aqua) can be acquired as the multi-channel to-be-recognized fluorescent signal point image, wherein each channel of the to-be-recognized fluorescent signal point image includes the imaging of the fluorescent signal points inside the same cell. In practice, the multi-channel fluorescent signal point image including multiple cell objects obtained by imaging the fluorescent signal points of the sample can be manually annotated to realize cell segmentation, or the cell segmentation can be automatically performed by a program, and then the multi-channel fluorescent signal point image of a certain cell can be obtained as the above-mentioned multi-channel to-be-recognized fluorescent signal point image.

[0060] And each channel of the to-be-recognized fluorescent signal point image is an image photographed by detecting multiple DNA sequences or chromosomes in the sample by using the fluorescent marker probe corresponding to the corresponding channel. Each fluorescent dye corresponds to a specific fluorescent marker probe, so that multiple gene or chromosome expression and positioning information can be analyzed simultaneously in one sample. For example, fluorescent dyes such as FITC, TRITC, Cy5, and Cy7 can be used, each dye corresponds to a channel, and is used to label different nucleic acid sequences, respectively.

[0061] In some embodiments, the to-be-recognized fluorescence signal point image of each channel can be a gray-scale image.

[0062] In step 202, image feature extraction is performed on the multi-channel to-be-recognized fluorescence signal point image based on the preliminary fusion feature extraction model, and a preliminary fusion feature map is obtained.

[0063] In this embodiment, the execution subject of the fluorescence signal point recognition method can perform image feature extraction on the multi-channel to-be-recognized fluorescence signal point image obtained in step 201 based on the preliminary fusion feature extraction model, and obtain a preliminary fusion feature map.

[0064] Here, the preliminary fusion feature extraction model is used to perform image feature extraction on the to-be-recognized fluorescence signal point image of each channel, and then fuse the channel image features extracted for the to-be-recognized fluorescence signal point images of each channel to obtain the preliminary fusion feature map. Thus, the image features of the to-be-recognized fluorescence signal point images of each channel are preliminarily learned in the preliminary fusion feature map.

[0065] For example, the preliminary fusion feature extraction model can include N channel feature extraction models, where N is the number of to-be-recognized fluorescence signal point images or the number of channels. Each channel corresponds to a channel feature extraction model. Then, the fluorescence signal point image of each channel can be input into the corresponding channel feature extraction model to obtain the channel image feature of the channel, and then the channel image features of each channel are fused to obtain the preliminary fusion feature map.

[0066] Optionally, step 202 can be performed as follows:

[0067] First, based on a plurality of consecutive depth separable convolution layers, image feature extraction is sequentially performed on the to-be-recognized fluorescence signal point images of each channel to obtain the channel image features of each channel output by the last layer of depth separable convolution layer.

[0068] Here, the convolution kernels in each layer of depth separable convolution layer are divided into multiple groups, and the number of groups of convolution kernels is the same as the number of channels corresponding to the multi-channel to-be-recognized fluorescence signal point image. Each group of convolution kernels is used to perform image feature extraction on the to-be-recognized fluorescence signal point image of the corresponding channel.

[0069] Then, feature fusion is performed on the spliced results of the channel image features of each channel to obtain the preliminary fusion feature map.

[0070] Here, various feature fusion methods can be used to fuse the spliced results of the channel image features of each channel to obtain a preliminary fusion feature map. For example, the spliced results of the channel image features of each channel can be directly used as the preliminary fusion feature map. Alternatively, the spliced results of the channel image features of each channel can be convolved to obtain the preliminary fusion feature map.

[0071] In step 203, multi-scale feature extraction is performed on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map.

[0072] The preliminary fusion feature map extracted in step 202 only completes the preliminary feature extraction and fusion of the to-be-identified fluorescence signal point images of each channel. To improve the accuracy of subsequent fluorescence signal point recognition, more complex features need to be learned. Therefore, multi-scale feature extraction can be performed on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a fusion feature map.

[0073] Here, the multi-scale feature extraction model is used to extract complex features from multiple scales. For example, the multi-scale feature extraction model can be various Backbone networks. For example, ResNet, VGG, Inception, Swin Transformer V2 T, etc.

[0074] In some optional embodiments, please refer to Figure 3 , Figure 3 A model structure diagram of one embodiment of the fluorescence signal point recognition method according to the present disclosure is shown. As shown in Figure 3 , the multi-scale feature extraction model can include a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an upsampling network, a splicing network, and a fourth C2f network connected in sequence, and the outputs of the second C2f network and the upsampling network are input into the splicing network.

[0075] Among them, the first C2f network, the second C2f network, the third C2f network and the fourth C2f network can all be “CSPBottleneck with 2 convolutions”, that is, a CSP (Cross Stage Partial) bottleneck structure with two convolution layers.

[0076] The first C2f network, the second C2f network, the third C2f network, and the fourth C2f network all introduce the idea of ELAN (Efficient Lightweight Attention Network), which is a lightweight attention mechanism that can help the model better capture important features in the image. ELAN is used to enhance the transmission of gradient flow information. By introducing ELAN, the first C2f network, the second C2f network, the third C2f network, and the fourth C2f network can better handle the details and contextual information in the image, thereby improving the accuracy and performance of the model.

[0077] The forward propagation process of the first C2f network, the second C2f network, the third C2f network, and the fourth C2f network first passes the input data through the first convolutional layer, then divides the output into two parts. One part is directly passed to the output, and the other part is processed through multiple Bottleneck networks. Finally, the results of the two parts are spliced in the channel dimension, and the final output is obtained through the second convolutional layer. This design helps to increase the non-linear ability and representation ability of the network, thereby improving the modeling ability of the network for complex data. At the same time, by splicing the features of different branches in the channel dimension, feature fusion is achieved, enriching the expression ability of the features.

[0078] The first convolutional neural network and the second convolutional neural network both include conv2d (two-dimensional convolutional layer), batchnorm2d (two-dimensional batch normalization layer), and SiLU (Sigmoid Linear Unit, also known as Swish activation function) network structure is a basic component of the convolutional neural network (CNN) commonly used in deep learning. Among them: conv2d (two-dimensional convolutional layer) is the core of the convolutional neural network, used to extract image features; batchnorm2d (two-dimensional batch normalization layer) reduces internal covariate shift by normalizing (normalizing) the mean and variance of each small batch of data, making the network training more stable, and is used to improve training speed, stability, and performance; SiLU (Sigmoid Linear Unit) self-gated activation function combines Sigmoid function and identity function, allowing the model to adaptively adjust the activation degree of each neuron.

[0079] The SPPF (Spatial Pyramid Pooling - Fast) network is a variant or fast version of the Spatial Pyramid Pooling (SPP) network, which can improve processing speed and efficiency while maintaining the ability to fuse multi-scale features. Specifically, SPPF includes the following steps:

[0080] 1. Feature map input: receiving a feature map from the third C2f network output as input.

[0081] 2. Fast pooling operation: using multiple small pooling layers (such as three 3x3 max pooling) to replace a single large core pooling operation to pool the input data, reducing the amount of calculation while maintaining the ability of multi-scale feature fusion.

[0082] 3. Feature splicing and output: after the fast pooling operation, the outputs of different pooling layers are spliced to form a fixed-length feature vector, which is output to the subsequent up-sampling network.

[0083] The SPPF network significantly improves the calculation speed by reducing the number and size of the pooling core. In addition, SPPF still retains the core advantage of SPP, that is, it can fuse feature information of different scales, which helps to improve the robustness of the model to object size changes. Finally, by optimizing the pooling strategy, SPPF reduces the memory requirement of the model while maintaining performance.

[0084] The up-sampling network is a key component of the multi-scale feature extraction model, which is used to enlarge the low-resolution feature map to a higher resolution. This process is particularly important for subsequent fluorescent signal point recognition, as signal points are often small in size, making it difficult for subsequent fluorescent signal point recognition. The up-sampling network can help the model better detect fluorescent signal points of different scales.

[0085] As an example, the up-sampling network can include but is not limited to Feature Pyramid Networks (FPN). In FPN, deep feature maps (with rich semantic information but low resolution) are fused with shallow feature maps (with high resolution but less semantic information) through up-sampling, generating a series of feature maps with different resolutions.

[0086] The feature map output by the up-sampling network and the feature map output by the second C2f are input to the splicing network for splicing and fusion to obtain the splicing and fusion feature. Specifically, the splicing network can first splice the feature map output by the up-sampling network and the feature map output by the second C2f, and then further process the spliced feature through a convolution layer to enhance the expression ability of the feature and obtain the splicing and fusion feature.

[0087] The above splicing and fusion feature is input to the fourth C2f network for further feature extraction to obtain the final multi-scale fusion feature map.

[0088] The global context information is effectively extracted in the multi-scale fusion feature map obtained by using the optional embodiment, which can better extract the signal point features, thereby improving the performance of the overall feature extraction, and can also realize feature dimension reduction to reduce the calculation amount and improve the calculation speed.

[0089] In step 204, the multi-scale fusion feature map is subjected to fluorescent signal point recognition based on a fluorescent signal point recognition model, to obtain a fluorescent signal point recognition result of each channel to-be-recognized fluorescent signal point image.

[0090] In the embodiment, the execution subject can input the multi-scale fusion feature map into the fluorescent signal point recognition model to obtain the fluorescent signal point recognition result of each channel to-be-recognized fluorescent signal point image.

[0091] Here, the fluorescent signal point recognition model is used to represent the corresponding relationship between the multi-scale fusion feature map and the fluorescent signal point recognition result of each channel. In practice, the fluorescent signal point recognition model can use various existing or future developed machine learning models, which are not limited in the present disclosure.

[0092] In some optional embodiments, the fluorescent signal point recognition result can include a signal point region information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model can include a signal point region information map extraction model and a signal point heat map extraction model. Accordingly, step 204 can include steps 2041 to 2043 as shown in Figure 4

[0093] In step 2041, the multi-scale fusion feature map is subjected to signal point region information map extraction based on the signal point region information map extraction model, to obtain a signal point region information map of each channel to-be-recognized fluorescent signal point image.

[0094] In step 2042, the multi-scale fusion feature map is subjected to signal point heat map extraction based on the signal point heat map extraction model, to obtain a signal point heat map of each channel to-be-recognized fluorescent signal point image.

[0095] It should be noted that step 2041 can be performed first and then step 2042, or step 2042 can be performed first and then step 2041, or steps 2041 and 2042 can be performed simultaneously, which are not limited in the present disclosure.

[0096] Here, for each channel, the signal point region information map of the channel to-be-recognized fluorescent signal point image is used to map each pixel point in the feature map of the preset feature map size of the channel to an image region in the channel to-be-recognized fluorescent signal point image.

[0097] ​It can be understood that, in order to reduce the model parameters and reduce the amount of calculation, in the process of inputting the multi-channel to-be-recognized fluorescent signal point image into the preliminary fusion feature extraction model and the multi-scale feature extraction model, the signal point heat map extraction model and the signal point region information map extraction model, feature extraction and dimension reduction will be performed on the input data, and the size of the signal point heat map and the signal point region information map output by the signal point heat map extraction model and the signal point region information map extraction model (i.e. the preset feature map size) should be as small as possible, and in most cases will be smaller than the image size of each channel to-be-recognized fluorescent signal point image. That is to say, if the signal point heat map and the signal point region information map are mapped to the to-be-recognized fluorescent signal point image of each channel, the region size of the mapped to-be-recognized fluorescent signal point image will also be larger than the image size of the signal point heat map and the signal point region information map. In order to locate the signal point region in the to-be-recognized fluorescent signal point image, each pixel point in the signal point heat map with the preset feature map size needs to be mapped into the to-be-recognized fluorescent signal point image through the signal point region information corresponding to the corresponding pixel point in the signal point region information map, and then the probability value of the corresponding image region in the mapped to-be-recognized fluorescent signal point image being a signal point region is determined.

[0098] Suppose that the multi-channel to-be-recognized fluorescent signal point image includes Red, Green, Gold and Aqua four-channel to-be-recognized fluorescent signal point images I re , I gr , I go , and I aq . After inputting the to-be-recognized fluorescent signal point images I re , I gr , I go , and I aq into the preliminary fusion feature extraction model and the multi-scale feature extraction model, a multi-scale fusion feature map FP is obtained. After inputting the multi-scale fusion feature map FP into the signal point region information map extraction model, the signal point region information map extraction of the multi-scale fusion feature map can be performed, and the signal point region information maps AI re , AI gr , AI go , and AI aq of the Red, Green, Gold and Aqua four-channel to-be-recognized fluorescent signal point images I re , I gr , I go , and I aq can be obtained respectively. The image sizes of the signal point region information maps AI re , AI gr , AI go , and AI aq are the same, and are all preset feature map sizes, for example, can be 80*80. Taking the Aqua channel as an example, for the signal point region information map AI aq(x aq , y aq ) of the pixel point, assuming that the region in the Aqua channel to-be-identified fluorescent signal point image I aq according to the signal point region information mapping of the (x aq , y aq ) pixel point is a region D aq with (x' aq , y' aq ) as the center point coordinate, a height of H and a width of W, then the signal point region information corresponding to the pixel point with coordinates (x aq , y aq ) in the signal point region information image AI aq can include the center point coordinates (x' aq , y' aq ), the height H and the width W in the Aqua channel to-be-identified fluorescent signal point image I aq .

[0099] In addition, by inputting the multi-scale fusion feature map into the signal point heat map extraction model, the signal point heat map of each channel to-be-identified fluorescent signal point image can be obtained. Here, the signal point heat map includes the probability of each pixel point in the feature map under the preset feature map size being a signal point region after mapping to the image region in the fluorescent signal point image according to the signal point region information image.

[0100] As an example, when including Red, Green, Gold and Aqua four channels, inputting the multi-scale fusion feature map into the signal point heat map extraction model can obtain four signal point heat maps HM re , HM gr , HM go and HM aq corresponding to the Red, Green, Gold and Aqua four channel to-be-identified fluorescent signal point images I re , I gr , I go and I aq respectively, the image sizes of the four signal point heat maps HM re , HM gr , HM go and HM aq are the same, and are all preset feature map sizes, for example, can be 80*80. The pixel value of each pixel point in the above four signal point heat maps HM re , HM gr , HM go and HM aq can be a probability value between 0-1. As an example, the signal point heat map HM aqThe pixel value of each pixel in the image is used to represent the regional information map AI of the pixel according to the corresponding channel (i.e. Aqua channel) aq Mapped to the fluorescent signal point image I to be identified aq The probability that the image area in the image is a signal point area. Take the signal point area information graph AI of the Aqua channel in the above example aq The median coordinate is (x aq , y aq ) After the pixel point is mapped to the signal point area information, it can be mapped to the area D in the Aqua channel fluorescence signal point image to be identified. aq , assuming that the signal point heat map HM aq The median coordinate is (x aq , y aq ) The probability value corresponding to the pixel point is P aq , then the Aqua channel fluorescence signal point image I aq Middle Area D aq The probability of being a signal point area is P aq .

[0101] Step 2043 : For each channel of the fluorescent signal point image to be identified, a fluorescent signal point identification result of the fluorescent signal point image to be identified in the channel is determined based on the signal point region information map and the signal point heat map of the fluorescent signal point image to be identified in the channel.

[0102] Here, the signal point area information map of the fluorescent signal point image to be identified in each channel can be used to locate the fluorescent signal point image area to be identified in the corresponding channel, and then the probability that the fluorescent signal point image area to be identified in the corresponding channel is the signal point area can be determined based on the signal point heat map of the fluorescent signal point image to be identified in the corresponding channel.

[0103] Finally, according to the probability that the region of the to-be-identified fluorescent signal point image of each channel is a signal point region, the fluorescent signal point recognition result of the to-be-identified fluorescent signal point image of the corresponding channel is determined.

[0104] Specifically, for each pixel point in the fluorescence signal point image area to be identified in each channel, the probability that the pixel point is a signal point area can be determined as the probability that the fluorescence signal point image area to be identified in the channel to which the pixel point belongs is a signal point area, and then the probability that each pixel point in the fluorescence signal point image to be identified in each channel is a signal point pixel point can be obtained.

[0105] Then, the probability that each pixel in the to-be-identified fluorescent signal point image of each channel is a signal point pixel can be used as the fluorescent signal point identification result.

[0106] Or also can be according to each channel to be identified fluorescent signal point image in each pixel point for signal point pixel point probability and the preset signal point probability threshold comparison result, generate for representing each channel to be identified fluorescent signal point image in each pixel point whether signal point pixel point signal point mask image, and above described signal point mask image as fluorescent signal point recognition result.

[0107] Or also, for each pixel point in the signal point heat map, the pixel point with a corresponding probability value greater than the preset signal point threshold is determined as a signal point heat map pixel point. Then, the pixel point in the signal point heat map with a corresponding probability value greater than the preset signal point threshold in the signal point region information map of each channel is determined as a signal point region pixel point, and then the signal point region pixel point is mapped to the signal point image region in the to-be-identified fluorescent signal point image. Finally, the region information of the mapped each to-be-identified fluorescent signal point image region is determined as the fluorescent signal point recognition result. For example, the fluorescent signal point recognition result can include the key point (the key point can be, for example, a center point or any vertex coordinate of a rectangle) position and region size information (for example, the height and width information of the signal point image region corresponding rectangle) of the signal point image region in each channel to-be-identified fluorescent signal point image.

[0108] In some optional embodiments, the signal point heat map extraction model can include a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0109] Here, the first image feature extraction network and the second image feature extraction network are used to extract image features step by step. The subsequent convolution layer is used to further extract features and reduce dimension of the features, and the final activation function layer is used to map the input data to a probability value between 0 and 1. For example, the activation function layer here can be a Sigmoid activation function.

[0110] As an example, the first image feature extraction network and the second image feature extraction network can both be CBL networks. The CBL network includes a convolution layer (Convolution Layer), a batch normalization layer (Batch Normalization Layer), and an activation function layer (Leaky ReLU Layer) connected in sequence.

[0111] In some optional embodiments, as shown in Figure 3 The signal point region information map extraction model can include a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model. Wherein:

[0112] The signal point width extraction model characterizes the correspondence between the multi-scale fusion feature map and the signal point area width map for each channel. The image size of the signal point area width map is the preset feature size (for example, 80*80), and the pixel value corresponding to each pixel in the signal point area width map is the width of the image area mapped to the fluorescent signal point image to be identified.

[0113] The signal point height extraction model characterizes the correspondence between the multi-scale fusion feature map and the signal point area height map for each channel. The image size of the signal point area height map is the preset feature size (for example, 80*80), and the pixel value corresponding to each pixel in the signal point area height map is the height of the image area mapped to the fluorescent signal point to be identified.

[0114] The signal point position horizontal coordinate offset extraction model is used to characterize the correspondence between the multi-scale fusion feature map and the signal point position horizontal coordinate offset map for each channel. The image size of the signal point position horizontal coordinate offset height map is a preset feature size (for example, 80*80). The pixel value corresponding to each pixel in the signal point position horizontal coordinate offset map is used to map the pixel coordinate to the horizontal coordinate of the key point in the image region of the fluorescent signal point to be identified.

[0115] The signal point position vertical coordinate offset extraction model is used to characterize the correspondence between the multi-scale fusion feature map and the signal point position vertical coordinate offset map for each channel. The image size of the signal point position vertical coordinate offset height map is a preset feature size (for example, 80*80). The pixel value corresponding to each pixel in the signal point position vertical coordinate offset map is used to map the pixel coordinate to the vertical coordinate of the key point in the image region of the fluorescent signal point to be identified.

[0116] Taking the Aqua channel as an example, for the signal point area information graph AI aq The median coordinate is (x aq , y aq ) pixel point, the coordinates in the signal point area width diagram are (x aq , y aq ) corresponds to the pixel value W, and the coordinates in the height map of the signal point area are (x aq , y aq ) corresponds to the pixel value H, and the coordinates of the signal point position in the horizontal coordinate offset diagram are (x aq , y aq ) pixel value Offset_W, the coordinates of the signal point position ordinate offset in the figure are (x aq , y aq ) corresponding to the pixel value Offset_H, which means that the coordinates of the above signal point area information diagram can beaq , y aq The region in which the pixel point of the Aqua channel is mapped to the region in the image of the to-be-recognized fluorescent signal point of the Aqua channel is a region D with (x aq + Offset W, y aq + Offset H) as the key point coordinates, a height of H, and a width of W. aq .

[0117] In some optional embodiments, the signal point width extraction model can include a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model can include a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model can include a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model can include a sixth image feature extraction network and a convolutional layer connected in sequence.

[0118] The third image feature extraction network, the fourth image feature extraction network, the fifth image feature extraction network, and the sixth image feature extraction network can each be the CBL network described above.

[0119] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model, and the fluorescent signal point recognition model can be obtained by pre-training through the training steps 500 as shown in FIG. 13: Figure 5

[0120] Step 501: Obtain a sample data set.

[0121] Here, the sample data can include multi-channel sample fluorescent signal point images and labeled fluorescent signal point recognition results of each channel sample fluorescent signal point image. The multi-channel sample fluorescent signal point image is a multi-channel fluorescent signal point image obtained by photographing the same field of view including the same cell object.

[0122] The labeled fluorescent signal point recognition result is used to represent the position or region (for example, a rectangular region) of the fluorescent point in the fluorescent signal point image. As an example, the labeled fluorescent signal point recognition result can include the four vertex coordinates of the rectangular region where the signal point is located, or one vertex coordinate and the width and height of the rectangular region, etc.

[0123] Step 502: Based on the sample data set, the preliminary fusion feature extraction model, the multi-scale feature extraction model, and the fluorescent signal point recognition model are trained.

[0124] ​Specifically, the multi-channel sample fluorescence signal point image in the sample data can be input into the preliminary fusion feature extraction model to obtain a preliminary fusion feature map; the preliminary fusion feature map can be input into the multi-scale feature extraction model to obtain a multi-scale fusion feature map; then, the multi-scale fusion feature map can be input into the fluorescence signal point recognition model to obtain a fluorescence signal point recognition result of each channel sample fluorescence signal point image; finally, the model parameters of the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescence signal point recognition model are adjusted based on the difference between the obtained fluorescence signal point recognition result of each channel sample fluorescence signal point image and the corresponding labeled fluorescence signal point recognition result, and then the trained preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescence signal point recognition model are obtained.

[0125] Here, various loss function calculation methods can be used to calculate the difference between the obtained fluorescence signal point recognition result of each channel sample fluorescence signal point image and the corresponding labeled fluorescence signal point recognition result. For example, Mean Squared Error (MSE), Mean Absolute Error (MAE), Cross-Entropy Loss, Logarithmic Loss, Huber Loss, etc.

[0126] Various parameter adjustment methods can also be used to adjust the model parameters of the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescence signal point recognition model. For example, one or a combination of at least two of Gradient Descent, Stochastic Gradient Descent (SGD), Mini-batch Gradient Descent, Nesterov Accelerated Gradient (NAG), AdaGrad, RMSProp, Adam (Adaptive Moment Estimation), AdaDelta, Nadam (Nesterov-accelerated Adaptive Moment Estimation), Learning Rate Decay, Early Stopping, Weight Decay, Dropout, and Batch Normalization, but not limited to these.

[0127] The fluorescence signal point recognition method provided by the above embodiments of the present disclosure comprises: performing image feature extraction on a plurality of channel fluorescence signal point images to be recognized including a same cell object based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; performing multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; and finally, performing fluorescence signal point recognition on the multi-scale fusion feature map based on a fluorescence signal point recognition model to obtain a fluorescence signal point recognition result of each channel fluorescence signal point image to be recognized. That is, only the multi-channel fluorescence signal point images photographed under the same field of view need to be input, and the fluorescence signal point recognition result of each channel fluorescence signal point image can be output. Thus, end-to-end fluorescence signal point recognition can be realized, fluorescence signal point recognition can be realized based on multi-channel fluorescence signal point images, and fluorescence signal point recognition can be completed with fewer model parameters, thereby saving the calculation amount and improving the calculation efficiency.

[0128] Embodiment 1:

[0129] In the specific experiment, the experimental data is derived from 82 samples, and 1 to 4 groups of four-channel (specifically including) fluorescence signal point images are extracted from each sample. The selected standard is that the cell density in each fluorescence signal point image is relatively uniform. Finally, 110 groups of multi-channel fluorescence signal point image sample data are formed, and each fluorescence signal point image is labeled with a fluorescence signal point recognition result by 20 labeling personnel, and reviewed by 5 senior labeling personnel. The above 110 samples are divided into a training set and a validation set. The training set is 60 groups of sample data (specifically including multi-channel fluorescence signal point images and corresponding labeled fluorescence signal point recognition results), and the test set is 50 groups of sample data.

[0130] The model training process is as follows:

[0131] 1) Preprocess the input image:

[0132] According to the cell contour labeling of the FISH image, the fluorescence signal point images of the four channels (Red, Green, Gold and Aqua four channels) are cut into cell-based signal point images, and converted into gray scale images. Then, the signal point images of the four channels are merged into one image with a dimension of (320, 320, 4).

[0133] 2) Data augmentation:

[0134] The following data augmentation is performed on the gray scale image after merging the four channels: random brightness enhancement, letterbox enhancement, HSV color enhancement, random disturbance enhancement (random perspective), gray scale and normalization.

[0135] 3) Training method:

[0136] The enhanced data is input into the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model as shown in Figure 3 The loss value between the output of the model and the labeled fluorescent signal point recognition result is calculated according to the loss function, Adam is used as the optimizer, the learning rate is 1.0e-3, weight_decay is 5e-4, and the parameters of the preliminary fusion feature extraction model, the multi-scale fusion extraction model and the fluorescent signal point recognition model are updated. ReduceLROnPlateau is used as lr_scheduler, factor is 0.9, patience is 5, and the learning rate of the optimizer is adjusted.

[0137] 100 epochs are trained, and the F1 score is used as the evaluation index. The sum of the F1 scores of the four-channel signal point images is used as the best model. The algorithm uses 0.3 as the threshold.

[0138] Loss function: The loss function of the signal point heat map extraction model uses focal loss, and the loss functions of the signal point width extraction model, the signal point height extraction model, the signal point position horizontal coordinate offset extraction model and the signal point position vertical coordinate offset extraction model all use reg_l1_loss.

[0139] The model inference process is as follows:

[0140] The four-channel fluorescent signal point image based on cells is merged into one image with a dimension of (1, 4, 320, 320), which is input into the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model to obtain the position and size of the signal point of the four-channel fluorescent signal point image based on cells.

[0141] The specific experimental results are shown in Table 1:

[0142] Table 1. Performance comparison table of algorithm

[0143]

[0144] Fnet in Table 1 corresponds to the experimental data of the fluorescent signal point recognition method disclosed in Chinese patent application 202210958372X, and FnetV2 is the experimental data of the specific experiment of the present disclosure.

[0145] FLOPS in Table 1 is the result obtained by the two methods for recognizing fluorescent signal points in a randomly generated test data.

[0146] In Table 1, the prediction time is the average prediction time of all data in all test sets.

[0147] The signal point detection needs to be completed for each channel fluorescence signal point image in Chinese patent application 202210958372X, and then joint training is completed for four-channel fluorescence signal point images, so as to complete the signal point detection of four-channel signal point images at the same time. The FnetV2 algorithm adopted by the present disclosure can read four-channel signal point images at the same time, extract features at the same time, and complete signal point detection of four-channel signal point images at the same time in the training process. As can be seen from Table 1, compared with Fnet, FnetV2 reduces the required storage space of model parameters, and reduces the calculation amount and improves the calculation speed.

[0148] Further referring to Figure 6 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of a fluorescence signal point recognition device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied in various electronic devices.

[0149] As shown in Figure 6 , the fluorescence signal point recognition device 600 of the present embodiment comprises an acquisition module 601, a preliminary feature fusion module 602, a multi-scale feature extraction module 603, and a fluorescence signal point recognition module 604. The acquisition module 601 is configured to acquire a multi-channel to-be-recognized fluorescence signal point image obtained by photographing the same field of view including the same cell object; the preliminary feature fusion module 602 is configured to perform image feature extraction on the multi-channel to-be-recognized fluorescence signal point image based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; the multi-scale feature extraction module 603 is configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; and the fluorescence signal point recognition module 604 is configured to perform fluorescence signal point recognition on the multi-scale fusion feature map based on a fluorescence signal point recognition model to obtain a fluorescence signal point recognition result of each channel to-be-recognized fluorescence signal point image.

[0150] In the present embodiment, the specific processing of the acquisition module 601, the preliminary feature fusion module 602, the multi-scale feature extraction module 603, and the fluorescence signal point recognition module 604 of the fluorescence signal point recognition device 600 and the technical effects brought by the same can be respectively referred to the related descriptions of step 201, step 202, step 203, and step 204 in the corresponding embodiment, which will not be repeated here. Figure 2

[0151] ​In some optional embodiments, the multi-scale feature extraction model comprises a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an up-sampling network, a concatenation network, and a fourth C2f network connected in sequence, and the output of the second C2f network and the up-sampling network is input into the concatenation network.

[0152] In some optional embodiments, the fluorescent signal point recognition result comprises a signal point region information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model comprises a signal point region information map extraction model and a signal point heat map extraction model.

[0153] The fluorescent signal point recognition module is further configured to:

[0154] perform signal point region information map extraction on the multi-scale fusion feature map based on the signal point region information map extraction model to obtain a signal point region information map of each channel of the to-be-recognized fluorescent signal point image, wherein the signal point region information map is used to map each pixel point in the feature map under the preset feature map size to an image region in the fluorescent signal point image;

[0155] perform signal point heat map extraction on the multi-scale fusion feature map based on the signal point heat map extraction model to obtain a signal point heat map of each channel of the to-be-recognized fluorescent signal point image, wherein the signal point heat map comprises a probability of each pixel point in the feature map under the preset feature map size being a signal point region after being mapped to an image region in the fluorescent signal point image according to the signal point region information map;

[0156] For each channel of the to-be-recognized fluorescent signal point image, determine a fluorescent signal point recognition result of the channel of the to-be-recognized fluorescent signal point image based on the signal point region information map and the signal point heat map of the channel of the to-be-recognized fluorescent signal point image.

[0157] In some optional embodiments, the signal point heat map extraction model comprises a first image feature extraction network, a second image feature extraction network, a convolutional layer, and an activation function layer connected in sequence.

[0158] In some optional embodiments, the first image feature extraction network and the second image feature extraction network each comprise a convolutional layer, a batch normalization layer, and an activation function layer connected in sequence.

[0159] In some optional embodiments, the signal point region information map extraction model comprises a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0160] In some optional embodiments, the signal point width extraction model comprises a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model comprises a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model comprises a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model comprises a sixth image feature extraction network and a convolutional layer connected in sequence.

[0161] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are obtained by pre-training through the following training steps:

[0162] Obtain a sample data set, wherein the sample data comprises multi-channel sample fluorescent signal point images and labeled fluorescent signal point recognition results of each channel sample fluorescent signal point image, and the multi-channel sample fluorescent signal point image is a multi-channel fluorescent signal point image obtained by photographing the same field of view comprising the same cell object;

[0163] Based on the sample data set, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are trained.

[0164] It should be noted that the implementation details and technical effects of each module in the fluorescent signal point recognition device provided by the embodiments of the present disclosure can refer to the descriptions of other embodiments of the present disclosure, which will not be described here.

[0165] Reference will be made below to Figure 7 which shows a structural schematic diagram of a computer system 700 suitable for implementing the electronic device of the present disclosure. Figure 7 The computer system 700 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0166] As shown in Figure 7 , the computer system 700 can include a processing device (such as a central processor, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the computer system 700 are also stored. The processing device 701, the ROM 702 and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0167] Generally, the following devices can be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, and the like; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 708 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 709. The communication devices 709 can allow the computer system 700 to communicate with other devices wirelessly or through wires to exchange data. Although Figure 7 The computer system 700 is shown with various devices, but it is understood that not all of the devices shown are required to implement or be present. More or fewer devices can alternatively be implemented or present.

[0168] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 709, or installed from the storage devices 708, or installed from the ROM 702. When the computer program is executed by the processing devices 701, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0169] It should be noted that the computer-readable medium in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, a RF (radio frequency) or the like, or any suitable combination of the above.

[0170] The computer-readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0171] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method of recognizing a fluorescent signal point as shown in the embodiments and optional implementation modes thereof. Figure 2 The method of recognizing a fluorescent signal point shown in the embodiments and optional implementation modes thereof.

[0172] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Python, Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0173] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0174] The modules described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the name of a module does not limit the module itself. For example, the obtaining module can also be described as a module that obtains a plurality of multi-channel to-be-identified fluorescence signal point images of the same field of view including the same cell object.

[0175] The above description is merely that of the preferred embodiments of the present disclosure and a description of the technical principles of the present disclosure. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions with the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the concept of the above disclosure. For example, the technical solutions formed by the mutual replacement of the above features and the technical features with similar functions disclosed in the present disclosure (but not limited to) without departing from the concept of the above disclosure.

Claims

1. A method for identifying fluorescent signal points, comprising: Acquire a multi-channel image of fluorescent signal points to be identified by photographing the same field of view including the same cell object; Based on the preliminary fusion feature extraction model, image feature extraction is performed on the multi-channel fluorescent signal point image to be identified, and then the channel image features extracted from the fluorescent signal point image to be identified in each channel are fused to obtain a preliminary fusion feature map; Based on a multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale fusion feature map; Based on the fluorescence signal point recognition model, fluorescence signal point recognition is performed on the multi-scale fusion feature map to obtain the fluorescence signal point recognition results of the fluorescence signal point image to be identified in each channel. The fluorescence signal point recognition results include a signal point area information map and a signal point heat map of a preset feature map size. The fluorescence signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model. The signal point area information map extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

2. The method of claim 1, wherein, The multi-scale feature extraction model includes a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an upsampling network, a splicing network and a fourth C2f network connected sequentially, and the outputs of the second C2f network and the upsampling network are input into the splicing network.

3. The method of claim 1, wherein, The fluorescent signal point recognition model is based on which the fluorescent signal point recognition is performed on the multi-scale fusion feature map to obtain the fluorescent signal point recognition result of the fluorescent signal point image to be recognized in each channel, including: Based on the signal point region information map extraction model, performing signal point region information map extraction on the multi-scale fusion feature map to obtain a signal point region information map of the fluorescence signal point image to be identified in each channel, wherein the signal point region information map is used to map each pixel point in the feature map at the preset feature map size to an image region in the fluorescence signal point image; Based on the signal point heat map extraction model, signal point heat map extraction is performed on the multi-scale fusion feature map to obtain a signal point heat map of the fluorescence signal point image to be identified in each channel, wherein the signal point heat map includes the probability that each pixel in the feature map at the preset feature map size is a signal point area after being mapped to the image area in the fluorescence signal point image according to the signal point area information map; For each channel of the fluorescent signal point image to be identified, the fluorescent signal point identification result of the fluorescent signal point image to be identified in the channel is determined based on the signal point area information map and the signal point heat map of the fluorescent signal point image to be identified in the channel.

4. The method of claim 3, wherein, The signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer and an activation function layer connected in sequence.

5. The method of claim 4, wherein, The first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

6. The method of claim 3, wherein, The signal point width extraction model comprises a third image feature extraction network and a convolution layer connected in sequence, the signal point height extraction model comprises a fourth image feature extraction network and a convolution layer connected in sequence, the signal point position horizontal coordinate offset extraction model comprises a fifth image feature extraction network and a convolution layer connected in sequence, and the signal point position vertical coordinate offset extraction model comprises a sixth image feature extraction network and a convolution layer connected in sequence.

7. The method of claim 1, wherein, The preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are obtained by the following training steps: obtain a sample data set, wherein the sample data comprises multi-channel sample fluorescent signal point images and labeled fluorescent signal point recognition results of each channel sample fluorescent signal point image, and the multi-channel sample fluorescent signal point image is a multi-channel fluorescent signal point image obtained by photographing the same field of view comprising the same cell object; based on the sample data set, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are trained.

8. A fluorescent signal point recognition device, comprising: an acquisition module configured to acquire a multi-channel to-be-recognized fluorescent signal point image obtained by photographing the same field of view comprising the same cell object; a preliminary fusion feature extraction module configured to perform image feature extraction on the multi-channel to-be-recognized fluorescent signal point image based on a preliminary fusion feature extraction model, and fuse channel image features extracted for each channel to-be-recognized fluorescent signal point image to obtain a preliminary fusion feature map; a multi-scale feature extraction module configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; a fluorescent signal point recognition module configured to perform fluorescent signal point recognition on the multi-scale fusion feature map based on a fluorescent signal point recognition model to obtain fluorescent signal point recognition results of each channel to-be-recognized fluorescent signal point image, wherein the fluorescent signal point recognition results comprise a signal point region information map and a signal point heat map of a preset feature map size, the fluorescent signal point recognition model comprises a signal point region information map extraction model and a signal point heat map extraction model, and the signal point region information map extraction model comprises a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model and a signal point position vertical coordinate offset extraction model.

9. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, wherein, The computer program is executed by one or more processors to implement the method of any one of claims 1-7.

11. A computer program product comprising computer programs / instructions, which are executed by a processor to implement the method of any one of claims 1-7.

Images