Fluorescence signal point identification method and related product

Abstract

The invention provides a fluorescent signal point identification method and a related product. According to one specific embodiment of the method, image feature extraction is performed on a multi-channel to-be-recognized fluorescence signal point image based on a preliminary fusion feature extraction model, and a preliminary fusion feature map is obtained; performing multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; and finally, based on a fluorescence signal point identification model, performing fluorescence signal point identification on the multi-scale fusion feature map to obtain a fluorescence signal point identification result of the to-be-identified fluorescence signal point image of each channel. The fluorescent signal point recognition result of the fluorescent signal point image of each channel can be output only by inputting the multi-channel fluorescent signal point images shot in the same visual field including the same cell object, and fluorescent signal point recognition is completed with fewer model parameters, so that the calculation amount is saved, and the calculation efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image recognition technology, and in particular to a fluorescent signal point recognition method and related products. Background Art

[0002] Images obtained based on fluorescence in situ hybridization (FISH) technology (including multi-channel fluorescence signal point images and Dapi images) can specifically detect diseases with high sensitivity. For example, circulating genetically abnormal cells (CAC) in lung cancer can be detected and judged one by one in the field of view through four-channel fluorescence signal point images and Dapi images.

[0003] Most of the current methods for automatically identifying CAC cells require the results of cell segmentation based on the Dapi image to segment the cells, locate each cell in the four-channel fluorescence signal point image and intercept the fluorescence signal point image of the corresponding cell in the corresponding channel, and then perform fluorescence signal point recognition on the intercepted fluorescence signal point image of each channel to obtain the fluorescence signal point recognition result of the corresponding channel. Finally, determine whether the cell is a CAC cell based on the number of fluorescence signal points in the cell. The important part of this process is to obtain the fluorescence signal point recognition results of each cell in the fluorescence signal point image of each channel. At present, the above process is divided into two stages: cell segmentation and fluorescence signal point recognition. However, the above two-stage method needs to locate each cell in the signal point image of the four channels, and then set the corresponding recognition model according to the signal point image of each cell in each channel for fluorescence signal point recognition, which leads to more model parameters and more storage space required, as well as a large amount of calculation and low calculation efficiency. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, an apparatus, an electronic device, a storage medium and a computer program product for identifying a fluorescent signal point.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for identifying fluorescent signal points, the method comprising: obtaining a multi-channel fluorescent signal point image to be identified by photographing the same field of view including the same cell object; performing image feature extraction on the multi-channel fluorescent signal point image to be identified based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; performing multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; performing fluorescent signal point identification on the multi-scale fusion feature map based on a fluorescent signal point identification model to obtain a fluorescent signal point identification result of the fluorescent signal point image to be identified in each channel.

[0006] In some optional embodiments, the multi-scale feature extraction model includes a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an upsampling network, a splicing network and a fourth C2f network connected sequentially, and the outputs of the second C2f network and the upsampling network are input into the splicing network.

[0007] In some optional embodiments, the fluorescent signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and

[0008] The method of performing fluorescence signal point recognition on the multi-scale fusion feature map based on the fluorescence signal point recognition model to obtain the fluorescence signal point recognition result of the fluorescence signal point image to be recognized in each channel includes:

[0009] Based on the signal point area information map extraction model, the signal point area information map is extracted from the multi-scale fusion feature map to obtain a signal point area information map of the fluorescence signal point image to be identified in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescence signal point image;

[0010] Based on the signal point heat map extraction model, the signal point heat map is extracted from the multi-scale fusion feature map to obtain a signal point heat map of the fluorescent signal point image to be identified in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map;

[0011] For each channel of the fluorescent signal point image to be identified, the fluorescent signal point identification result of the fluorescent signal point image to be identified in the channel is determined based on the signal point area information map and the signal point heat map of the fluorescent signal point image to be identified in the channel.

[0012] In some optional embodiments, the signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0013] In some optional embodiments, the first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

[0014] In some optional implementations, the signal point area information graph extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0015] In some optional embodiments, the signal point width extraction model includes a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model includes a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model includes a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model includes a sixth image feature extraction network and a convolutional layer connected in sequence.

[0016] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are pre-trained through the following training steps:

[0017] Acquire a sample data set, wherein the sample data includes a multi-channel sample fluorescence signal point image and annotated fluorescence signal point recognition results of the sample fluorescence signal point image of each channel, wherein the multi-channel sample fluorescence signal point image is a multi-channel fluorescence signal point image obtained by photographing the same field of view including the same cell object;

[0018] Based on the sample data set, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are trained and obtained.

[0019] In a second aspect, an embodiment of the present disclosure provides a fluorescence signal point identification device, which includes: an acquisition module, configured to acquire a multi-channel fluorescence signal point image to be identified by photographing the same field of view including the same cell object; a feature preliminary fusion module, configured to perform image feature extraction on the multi-channel fluorescence signal point image to be identified based on a preliminary fusion feature extraction model, to obtain a preliminary fusion feature map; a multi-scale feature extraction module, configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model, to obtain a multi-scale fusion feature map; a fluorescence signal point identification module, configured to perform fluorescence signal point identification on the multi-scale fusion feature map based on a fluorescence signal point identification model, to obtain a fluorescence signal point identification result of the fluorescence signal point image to be identified in each channel.

[0020] In some optional embodiments, the multi-scale feature extraction model includes a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an upsampling network, a splicing network and a fourth C2f network connected sequentially, and the outputs of the second C2f network and the upsampling network are input into the splicing network.

[0021] In some optional embodiments, the fluorescent signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and

[0022] The fluorescent signal point recognition module is further configured as follows:

[0023] Based on the signal point area information map extraction model, the signal point area information map is extracted from the multi-scale fusion feature map to obtain a signal point area information map of the fluorescence signal point image to be identified in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescence signal point image;

[0024] Based on the signal point heat map extraction model, the signal point heat map is extracted from the multi-scale fusion feature map to obtain a signal point heat map of the fluorescent signal point image to be identified in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map;

[0025] For each channel of the fluorescent signal point image to be identified, the fluorescent signal point identification result of the fluorescent signal point image to be identified in the channel is determined based on the signal point area information map and the signal point heat map of the fluorescent signal point image to be identified in the channel.

[0026] In some optional embodiments, the signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0027] In some optional embodiments, the first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

[0028] In some optional implementations, the signal point area information graph extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0029] In some optional embodiments, the signal point width extraction model includes a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model includes a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model includes a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model includes a sixth image feature extraction network and a convolutional layer connected in sequence.

[0030] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are pre-trained through the following training steps:

[0031] Acquire a sample data set, wherein the sample data includes a multi-channel sample fluorescence signal point image and annotated fluorescence signal point recognition results of the sample fluorescence signal point image of each channel, wherein the multi-channel sample fluorescence signal point image is a multi-channel fluorescence signal point image obtained by photographing the same field of view including the same cell object;

[0032] Based on the sample data set, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are trained and obtained.

[0033] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0034] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner in the first aspect.

[0035] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program / instruction, which implements the method described in any implementation manner in the first aspect when the computer program / instruction is executed by a processor.

[0036] In order to solve the problem that the current two-stage fluorescence signal point recognition method requires many model parameters and thus has low computational efficiency, the embodiments of the present disclosure provide a fluorescence signal point recognition method, device, electronic device, storage medium and computer program product. Based on the preliminary fusion feature extraction model, the multi-channel fluorescence signal point images to be identified for the same cell object are merged as a whole to extract image features, and a preliminary fusion feature map is obtained; then based on the multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale fusion feature map; finally, based on the fluorescence signal point recognition model, the multi-scale fusion feature map is recognized for the fluorescence signal point, and the fluorescence signal point recognition result of the fluorescence signal point image to be identified in each channel is obtained. That is, only the multi-channel fluorescence signal point images taken in the same field of view of a cell object after cell segmentation need to be input, and the fluorescence signal point recognition result of the fluorescence signal point image of each channel can be output. Thus, end-to-end fluorescence signal point recognition can be achieved, and fluorescence signal point recognition can be achieved based on multi-channel fluorescence signal point images, and fluorescence signal point recognition can be completed with fewer model parameters, saving the amount of calculation and improving the computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0038] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0039] Figure 2 is a flow chart of an embodiment of a fluorescent signal point identification method according to the present disclosure;

[0040] Figure 3 is a schematic diagram of a model structure of an embodiment of a fluorescent signal point recognition method according to the present disclosure;

[0041] Figure 4 is a decomposed flow chart of one embodiment of step 204 according to the present disclosure;

[0042] Figure 5 is a flow chart of one embodiment of the training step 500 according to the present disclosure;

[0043] Figure 6 is a schematic structural diagram of an embodiment of a fluorescent signal point recognition device according to the present disclosure;

[0044] Figure 7 It is a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the fluorescent signal point identification method, apparatus, electronic device, and storage medium of the present disclosure can be applied.

[0048] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0049] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as fluorescent signal point recognition applications, short video social applications, audio and video conferencing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0050] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with information input devices (e.g., keyboard, mouse, touch screen, microphone, camera, etc.) and information output devices (e.g., display screen, speaker, etc.), including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, used to provide fluorescent signal point recognition services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0051] In some cases, the fluorescent signal point identification method provided by the present disclosure may be executed by the terminal devices 101, 102, 103, and accordingly, the fluorescent signal point identification device may be disposed in the terminal devices 101, 102, 103. In this case, the system architecture 100 may not include the server 105.

[0052] In some cases, the fluorescent signal point identification method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining a multi-channel fluorescent signal point image to be identified by photographing the same field of view including the same cell object" can be performed by the terminal devices 101, 102, 103, and the steps of "extracting image features of the multi-channel fluorescent signal point image to be identified based on the preliminary fusion feature extraction model to obtain a preliminary fusion feature map" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the fluorescent signal point identification device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0053] In some cases, the fluorescent signal point identification method provided in the present disclosure may be executed by the server 105 , and accordingly, the fluorescent signal point identification device may also be set in the server 105 . In this case, the system architecture 100 may not include the terminal devices 101 , 102 , and 103 .

[0054] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0055] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0056] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a fluorescent signal point recognition method according to the present disclosure, the fluorescent signal point recognition method comprises the following steps:

[0057] Step 201 : acquiring a multi-channel image of fluorescent signal points to be identified by photographing the same field of view including the same cell object.

[0058] In this embodiment, the execution subject of the fluorescent signal point recognition method (for example Figure 1 The terminal device shown in the figure can obtain locally or remotely from other electronic devices connected to the execution subject network the multi-channel fluorescence signal point images to be identified obtained by shooting the same field of view including the same cell object.

[0059] Here, a multi-channel fluorescence signal point image (for example, a fluorescence image captured under four channels, Red, Green, Gold and Aqua) obtained by photographing the same field of view including the same cell object under a microscope can be obtained as a multi-channel fluorescence signal point image to be identified, wherein the fluorescence signal point image to be identified of each channel includes the fluorescence signal point imaging inside the same cell. In practice, cell segmentation can be achieved by manually annotating the multi-channel fluorescence signal point image including multiple cell objects obtained by performing fluorescence signal point imaging on the sample, or by automatically performing cell segmentation through a program, thereby obtaining a multi-channel fluorescence signal point image of one of the cells as the above-mentioned multi-channel fluorescence signal point image to be identified.

[0060] The image of the fluorescent signal point to be identified in each channel is an image taken by using the fluorescent labeled probe corresponding to the corresponding channel to detect multiple DNA sequences or chromosomes in the sample. Each fluorescent dye corresponds to a specific fluorescent labeled probe, so that the expression and location information of multiple genes or chromosomes can be analyzed simultaneously in one sample. For example, fluorescent dyes such as FITC, TRITC, Cy5 and Cy7 can be used, and each dye corresponds to a channel, which is used to label different nucleic acid sequences.

[0061] In some embodiments, the image of the fluorescent signal points to be identified in each channel may be a grayscale image.

[0062] Step 202: Based on the preliminary fusion feature extraction model, image features are extracted from the multi-channel fluorescent signal point images to be identified to obtain a preliminary fusion feature map.

[0063] In this embodiment, the execution body of the fluorescence signal point recognition method can extract image features from the multi-channel fluorescence signal point image to be recognized obtained in step 201 based on the preliminary fusion feature extraction model to obtain a preliminary fusion feature map.

[0064] Here, the preliminary fusion feature extraction model is used to extract image features of the fluorescent signal point images to be identified in each channel, and then fuse the channel image features extracted from the fluorescent signal point images to be identified in each channel to obtain a preliminary fusion feature map. Therefore, the image features of the fluorescent signal point images to be identified in each channel are preliminarily learned in the preliminary fusion feature map.

[0065] For example, the preliminary fusion feature extraction model may include N channel feature extraction models, where N is the number of fluorescent signal point images to be identified, or the number of channels. Each channel corresponds to a channel feature extraction model. Furthermore, the fluorescent signal point image of each channel can be input into the corresponding channel feature extraction model to obtain the channel image features of the channel, and then the channel image features of each channel are fused to obtain a preliminary fusion feature map.

[0066] Optionally, step 202 may be performed as follows:

[0067] Firstly, based on multiple consecutive depthwise separable convolutional layers, image features of the fluorescent signal point images to be identified in each channel are extracted in turn, and the channel image features of each channel output by the last depthwise separable convolutional layer are obtained.

[0068] Here, the convolution kernels in each depth-separable convolution layer are divided into multiple groups, the number of convolution kernel groups is the same as the number of channels corresponding to the multi-channel fluorescent signal point images to be identified, and each group of convolution kernels is used to extract image features of the fluorescent signal point images to be identified in the corresponding channel.

[0069] Then, feature fusion is performed on the splicing results of the channel image features of each channel to obtain a preliminary fused feature map.

[0070] Here, various feature fusion methods can be used to perform feature fusion on the splicing results of the channel image features of each channel to obtain a preliminary fusion feature map. For example, the splicing results of the channel image features of each channel can be directly used as the preliminary fusion feature map. Optionally, the splicing results of the channel image features of each channel can also be convolved to obtain a preliminary fusion feature map.

[0071] Step 203: Based on the multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale fusion feature map.

[0072] The preliminary fusion feature map extracted in step 202 only completes the preliminary feature extraction and fusion of the fluorescence signal point images to be identified in each channel. In order to improve the accuracy of subsequent fluorescence signal point identification, more complex features need to be learned. Therefore, multi-scale feature extraction can be performed on the preliminary fusion feature map based on the multi-scale feature extraction model to obtain a fusion feature map.

[0073] Here, the multi-scale feature extraction model is used to extract complex features from multiple scales. For example, the multi-scale feature extraction model can be various backbone networks. For example, ResNet, VGG, Inception, Swin Transformer V2 T, etc.

[0074] In some optional implementations, please refer to Figure 3 , Figure 3 FIG. 2 shows a schematic diagram of a model structure of an embodiment of a fluorescent signal point recognition method according to the present disclosure. Figure 3 As shown, the multi-scale feature extraction model may include a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an upsampling network, a splicing network and a fourth C2f network connected sequentially, and the outputs of the second C2f network and the upsampling network are input into the splicing network.

[0075] Among them, the first C2f network, the second C2f network, the third C2f network and the fourth C2f network can all be CSP bottleneck with 2convolutions", that is, a CSP (Cross Stage Partial) bottleneck structure with two convolutional layers.

[0076] The idea of ​​ELAN (Efficient Lightweight Attention Network) is introduced in the first C2f network, the second C2f network, the third C2f network, and the fourth C2f network. ELAN is a lightweight attention mechanism that can help the model better capture important features in the image. ELAN is used to enhance the transmission of gradient flow information. By introducing ELAN, the first C2f network, the second C2f network, the third C2f network, and the fourth C2f network can better process the details and context information in the image, thereby improving the accuracy and performance of the model.

[0077] The forward propagation process of the first C2f network, the second C2f network, the third C2f network, and the fourth C2f network first passes the input data through the first convolution layer, and then divides the output into two parts. One part is directly passed to the output, and the other part is processed by multiple Bottleneck networks. Finally, the results of the two parts are spliced ​​in the channel dimension and passed through the second convolution layer to obtain the final output. This design helps to increase the nonlinear ability and representation ability of the network, thereby improving the network's ability to model complex data. At the same time, by splicing the features of different branches in the channel dimension, feature fusion is achieved, enriching the expression ability of the features.

[0078] The first convolutional neural network and the second convolutional neural network both include the network structure of conv2d (two-dimensional convolutional layer), batchnorm2d (two-dimensional batch normalization layer) and SiLU (Sigmoid Linear Unit, also known as Swish activation function), which is the basic component of the common convolutional neural network (CNN) in deep learning. Among them: conv2d (two-dimensional convolutional layer) is the core of the convolutional neural network, which is used to extract image features; batchnorm2d (two-dimensional batch normalization layer) reduces the internal covariate shift by normalizing the mean and variance of each small batch of data, making the network training more stable, and is used to improve the training speed, stability and performance; SiLU (Sigmoid Linear Unit) self-gating activation function allows the model to adaptively adjust the activation level of each neuron by combining the Sigmoid function and the identity function.

[0079] The SPPF (Spatial Pyramid Pooling-Fast) network is a variant or fast version of the spatial pyramid pooling (Spatial Pyramid Pooling, referred to as SPP), which can improve processing speed and efficiency while maintaining the ability to fuse multi-scale features. Specifically, SPPF includes the following steps:

[0080] 1. Feature map input: Receive the feature map output from the third C2f network as input.

[0081] 2. Fast pooling operation: Multiple small pooling layers (such as three layers of 3x3 maximum pooling) are used to replace the pooling operation of a single large core to perform pooling operations on the input data to reduce the amount of calculation while maintaining the ability to fuse multi-scale features.

[0082] 3. Feature concatenation and output: After the fast pooling operation, the outputs of different pooling layers are concatenated to form a feature vector of fixed length, which is output to the subsequent upsampling network.

[0083] The SPPF network significantly improves the computation speed by reducing the number and size of pooling kernels. In addition, since SPPF still retains the core advantage of SPP, that is, it can fuse feature information of different scales, it helps to improve the robustness of the model to object scale changes. Finally, by optimizing the pooling strategy, SPPF reduces the memory requirements of the model while maintaining performance.

[0084] The upsampling network is a key component in the multi-scale feature extraction model, which is used to enlarge the low-resolution feature map to a higher resolution. This process is particularly important for the subsequent identification of fluorescent signal points. Since the signal points are often small in size, it is not conducive to the subsequent identification of fluorescent signal points. The upsampling network can help the model better detect fluorescent signal points of different scales.

[0085] As an example, the upsampling network may include, but is not limited to, Feature Pyramid Networks (FPN). In FPN, deep feature maps (with rich semantic information but low resolution) are fused with shallow feature maps (with high resolution but less semantic information) through upsampling, thereby generating a series of feature maps with different resolutions.

[0086] The feature map output by the upsampling network and the feature map output by the second C2f are input to the splicing network for splicing and fusion to obtain splicing and fusion features. Specifically, the splicing network can first splice the feature map output by the upsampling network and the feature map output by the second C2f, and then further process the spliced ​​features through a convolutional layer to enhance the expressiveness of the features and obtain splicing and fusion features.

[0087] The above-mentioned spliced ​​fusion features are then input into the fourth C2f network for further feature extraction to obtain the final multi-scale fusion feature map.

[0088] The multi-scale fusion feature map obtained by adopting this optional implementation method effectively extracts global context information, which can not only better extract signal point features, thereby improving the performance of overall feature extraction, but also achieve feature dimensionality reduction to reduce the amount of calculation and improve the calculation speed.

[0089] Step 204 , based on the fluorescence signal point recognition model, the fluorescence signal point recognition is performed on the multi-scale fusion feature map to obtain the fluorescence signal point recognition result of the fluorescence signal point image to be recognized in each channel.

[0090] In this embodiment, the above-mentioned execution subject can input the multi-scale fusion feature map into the fluorescent signal point recognition model to obtain the fluorescent signal point recognition result of the fluorescent signal point image to be recognized in each channel.

[0091] Here, the fluorescence signal point recognition model is used to characterize the correspondence between the multi-scale fusion feature map and the fluorescence signal point recognition results of each channel. In practice, the fluorescence signal point recognition model can adopt various existing or future developed machine learning models, which are not specifically limited in this disclosure.

[0092] In some optional embodiments, the fluorescent signal point recognition result may include a signal point area information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model may include a signal point area information map extraction model and a signal point heat map extraction model. Accordingly, step 204 may include the following: Figure 4 Steps 2041 to 2043 shown:

[0093] Step 2041, based on the signal point region information map extraction model, perform signal point region information map extraction on the multi-scale fusion feature map to obtain the signal point region information map of the fluorescent signal point image to be identified in each channel.

[0094] Step 2042: Based on the signal point heat map extraction model, perform signal point heat map extraction on the multi-scale fusion feature map to obtain a signal point heat map of the fluorescent signal point image to be identified in each channel.

[0095] It should be noted that, here, step 2041 may be executed first and then step 2042, or step 2042 may be executed first and then step 2041, or step 2041 and step 2042 may be executed simultaneously, and the present disclosure does not make any specific limitation on this.

[0096] Here, for each channel, the signal point area information map of the fluorescent signal point image to be identified in the channel is used to map each pixel point in the feature map under the preset feature map size of the channel to the image area in the fluorescent signal point image to be identified in the channel.

[0097] It is understandable that in order to reduce model parameters and reduce the amount of calculation, in the process of inputting the preliminary fusion feature extraction model and the multi-scale feature extraction model, the signal point heat map extraction model and the signal point area information map extraction model into the multi-channel fluorescent signal point image to be identified, the input data will be subjected to feature extraction and dimension reduction, and the signal point heat map and signal point area information map size (i.e., the preset feature map size) output by the signal point heat map extraction model and the signal point area information map extraction model should be as small as possible, and in most cases will be smaller than the image size of the fluorescent signal point image to be identified in each channel. In other words, if the signal point heat map and the signal point area information map are mapped to the fluorescent signal point image to be identified in each channel, the area size of the mapped fluorescent signal point image to be identified will also be larger than the image size of the signal point heat map and the signal point area information map. In order to locate the signal point area in the fluorescent signal point image to be identified, each pixel in the signal point heat map of the preset feature map size needs to be mapped to the fluorescent signal point image to be identified through the signal point area information corresponding to the corresponding pixel in the signal point area information map, and then the probability value of the corresponding image area in the mapped fluorescent signal point image to be identified is determined.

[0098] Assume that the multi-channel fluorescence signal point image to be identified includes the fluorescence signal point images to be identified of four channels: Red, Green, Gold and Aqua. re ,I gr ,I go and I aq . The fluorescent signal point image I to be identified re ,I gr ,I go and I aq After inputting the preliminary fusion feature extraction model and the multi-scale feature extraction model, the multi-scale fusion feature map FP is obtained. The multi-scale fusion feature map FP is input into the signal point area information map extraction model, and the signal point area information map is extracted from the multi-scale fusion feature map, and the four channels of red, green, gold and aqua fluorescent signal point images to be identified I can be obtained respectively. re ,I gr ,I go and I aq Signal point area information map AI re 、AI gr 、AI go and AI aq , signal point area information map AI re 、AI gr 、AI go and AI aq The image size is the same as the preset feature map size, for example, it can be 80*80. Taking the Aqua channel as an example, for the signal point area information map AI aqThe median coordinate is (x aq ,y aq ) pixels, assuming that according to (x aq ,y aq ) The signal point area information of the pixel is mapped to the fluorescent signal point image I to be identified in the Aqua channel aq The area in (x' aq ,y' aq ) is the center point coordinate, the height is H, and the width is W of the area D aq , then the signal point area information graph AI aq The median coordinate is (x aq ,y aq The signal point area information corresponding to the pixel point of the Aqua channel may include the fluorescent signal point image I to be identified. aq The center point coordinates (x' aq ,y' aq ), height H and width W.

[0099] In addition, by inputting the multi-scale fusion feature map into the signal point heat map extraction model, a signal point heat map of the fluorescence signal point image to be identified in each channel can be obtained. Here, the signal point heat map includes the probability that each pixel in the feature map at the preset feature map size is a signal point area after being mapped to the image area in the fluorescence signal point image according to the signal point area information map.

[0100] As an example, when four channels, Red, Green, Gold and Aqua, are included, the multi-scale fusion feature map is input into the signal point heat map extraction model, and the fluorescence signal point image I to be identified with the four channels of Red, Green, Gold and Aqua can be obtained. re ,I gr ,I go and I aq The four corresponding signal point heat maps HM re 、HM gr 、HM go and H.M. aq , four signal point heat maps HM re 、HM gr 、HM go and H.M. aq The image sizes of the four signal point heat maps HM are the same, which are all preset feature map sizes, for example, 80*80. re 、HM gr 、HM go and H.M. aq The pixel value of each pixel in can be a probability value between 0 and 1. As an example, the signal point heat map HM aqThe pixel value of each pixel in the image is used to represent the regional information map AI of the pixel according to the corresponding channel (i.e. Aqua channel) aq Mapped to the fluorescent signal point image I to be identified aq The probability that the image area in the image is a signal point area. Take the signal point area information graph AI of the Aqua channel in the above example aq The median coordinate is (x aq ,y aq ) The pixel point can be mapped to the area D in the Aqua channel fluorescent signal point image to be identified after the signal point area information is mapped. aq , assuming that the signal point heat map HM aq The median coordinate is (x aq ,y aq ) is the probability value corresponding to the pixel point aq , then the Aqua channel fluorescence signal point image I aq Middle Area D aq The probability of being a signal point area is P aq .

[0101] Step 2043, for each channel of the fluorescent signal point image to be identified, based on the signal point area information map and the signal point heat map of the fluorescent signal point image to be identified in the channel, determine the fluorescent signal point identification result of the fluorescent signal point image to be identified in the channel.

[0102] Here, the fluorescent signal point image area to be identified of the corresponding channel can be located through the signal point area information map of the fluorescent signal point image to be identified in each channel, and then the probability that the fluorescent signal point image area to be identified of the corresponding channel is the signal point area can be determined based on the signal point heat map of the fluorescent signal point image to be identified in the corresponding channel.

[0103] Finally, according to the probability that the to-be-identified fluorescent signal point image region of each channel is a signal point region, the fluorescent signal point recognition result of the to-be-identified fluorescent signal point image of the corresponding channel is determined.

[0104] Specifically, for each pixel in the fluorescent signal point image area to be identified in each channel, the probability that the pixel is a signal point area can be first determined as the probability that the fluorescent signal point image area to be identified in the channel to which the pixel belongs is a signal point area, and then the probability that each pixel in the fluorescent signal point image to be identified in each channel is a signal point pixel can be obtained.

[0105] Then, the probability that each pixel in the to-be-identified fluorescent signal point image of each channel is a signal point pixel can be used as the fluorescent signal point identification result.

[0106] Alternatively, a signal point mask image can be generated based on the comparison result of the probability that each pixel in the fluorescence signal point image to be identified in each channel is a signal point pixel and a preset signal point probability threshold, and the signal point mask image can be used as the fluorescence signal point identification result.

[0107] Alternatively, for each pixel in the signal point heat map, the pixel whose corresponding probability value is greater than the preset signal point threshold is determined as the signal point heat map pixel. Then, the pixel whose probability value corresponding to the pixel in the signal point heat map in each channel is greater than the preset signal point threshold is determined as the signal point area pixel, and then the signal point area pixel is mapped to the signal point image area in the fluorescent signal point image to be identified, and finally the area information of each mapped fluorescent signal point image area to be identified is determined as the fluorescent signal point recognition result. For example, the fluorescent signal point recognition result may include the key point position (the key point may be, for example, the center point or any vertex coordinate of a rectangle) of the signal point image area in the fluorescent signal point image to be identified in each channel and the area size information (for example, the height and width information of the rectangle corresponding to the signal point image area).

[0108] In some optional embodiments, the signal point heat map extraction model may include a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0109] Here, the first image feature extraction network and the second image feature extraction network are used to gradually extract image features. The subsequent convolutional layer is used to further extract features and feature dimensionality reduction, and the final activation function layer is used to map the input data to a probability value between 0 and 1. For example, the activation function layer here can be a Sigmoid activation function.

[0110] As an example, both the first image feature extraction network and the second image feature extraction network may be CBL networks. The CBL network includes a convolution layer, a batch normalization layer, and an activation function layer (Leaky ReLU Layer) connected in sequence.

[0111] In some optional embodiments, such as Figure 3 As shown, the signal point area information graph extraction model may include a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model. Among them:

[0112] The signal point width extraction model is used to characterize the correspondence between the multi-scale fusion feature map and the signal point area width map of each channel. The image size of the signal point area width map is a preset feature size (e.g., 80*80), and the pixel value corresponding to each pixel in the signal point area width map is the width of the image area mapped to the fluorescent signal point image to be identified by the pixel.

[0113] The signal point height extraction model is used to characterize the correspondence between the multi-scale fusion feature map and the signal point area height map of each channel. The image size of the signal point area height map is a preset feature size (e.g., 80*80), and the pixel value corresponding to each pixel in the signal point area height map is the height of the image area mapped to the fluorescent signal point image to be identified.

[0114] The signal point position horizontal coordinate offset extraction model is used to characterize the correspondence between the multi-scale fusion feature map and the signal point position horizontal coordinate offset map of each channel. The image size of the signal point position horizontal coordinate offset height map is a preset feature size (e.g., 80*80), and the pixel value corresponding to each pixel point in the signal point position horizontal coordinate offset map is used to map the pixel point coordinate to the key point horizontal coordinate of the image area in the fluorescent signal point image to be identified.

[0115] The signal point position ordinate offset extraction model is used to characterize the correspondence between the multi-scale fusion feature map and the signal point position ordinate offset map of each channel. The image size of the signal point position ordinate offset height map is a preset feature size (e.g., 80*80), and the pixel value corresponding to each pixel point in the signal point position ordinate offset map is used to map the pixel point coordinates to the key point ordinate coordinates of the image area in the fluorescent signal point image to be identified.

[0116] Taking the Aqua channel as an example, for the signal point area information graph AI aq The median coordinate is (x aq ,y aq ) pixel point, the coordinates in the signal point area width map are (x aq ,y aq ) corresponds to a pixel value W, and the coordinates in the signal point area height map are (x aq ,y aq ) corresponds to a pixel value of H, and the coordinates of the signal point position in the horizontal coordinate offset diagram are (x aq ,y aq ) corresponds to the pixel value Offset_W, and the coordinates of the signal point position ordinate offset in the figure are (x aq ,y aq ) corresponding to the pixel value Offset_H, indicating that the pixel with the coordinate (x aq,y aq ) is mapped to the area of ​​the Aqua channel fluorescence signal point image to be identified as (x aq +Offset_W,y aq +Offset_H) is the key point coordinate, the height is H, the width is W of the area D aq .

[0117] In some optional embodiments, the signal point width extraction model may include a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model may include a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model may include a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model may include a sixth image feature extraction network and a convolutional layer connected in sequence.

[0118] Among them, the third image feature extraction network, the fourth image feature extraction network, the fifth image feature extraction network and the sixth image feature extraction network can all be the above-mentioned CBL network.

[0119] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model can be obtained by Figure 5 The training steps 500 shown are pre-trained:

[0120] Step 501: Obtain a sample data set.

[0121] Here, the sample data may include a multi-channel sample fluorescence signal point image and annotated fluorescence signal point recognition results of each channel sample fluorescence signal point image. The multi-channel sample fluorescence signal point image is a multi-channel fluorescence signal point image obtained by photographing the same field of view including the same cell object.

[0122] The annotated fluorescent signal point recognition result is used to characterize the location or area (e.g., rectangular area) of the fluorescent point in the fluorescent signal point image. As an example, the annotated fluorescent signal point recognition result may include the coordinates of four vertices of the rectangular area where the signal point is located, or the coordinates of a vertex and the width and height of the rectangular area.

[0123] Step 502: Based on the sample data set, a preliminary fusion feature extraction model, a multi-scale feature extraction model and a fluorescent signal point recognition model are trained.

[0124] Specifically, the multi-channel sample fluorescence signal point image in the sample data can be input into the preliminary fusion feature extraction model to obtain a preliminary fusion feature map; the preliminary fusion feature map is then input into the multi-scale feature extraction model to obtain a multi-scale fusion feature map; then, the multi-scale fusion feature map is input into the fluorescence signal point recognition model to obtain the fluorescence signal point recognition results of each channel sample fluorescence signal point image; finally, based on the difference between the obtained fluorescence signal point recognition results of each channel sample fluorescence signal point image and the corresponding labeled fluorescence signal point recognition results, the model parameters of the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescence signal point recognition model are adjusted to obtain the trained preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescence signal point recognition model.

[0125] Here, various loss function calculation methods can be used to calculate the difference between the obtained fluorescence signal point recognition result of each channel sample fluorescence signal point image and the corresponding labeled fluorescence signal point recognition result, for example, Mean Squared Error (MSE), Mean Absolute Error (MAE), Cross-Entropy Loss, Logarithmic Loss, Huber Loss, etc.

[0126] Various parameter adjustment methods may also be used to adjust the model parameters of the preliminary fusion feature extraction model, the multi-scale feature extraction model, and the fluorescent signal point recognition model, including but not limited to one or a combination of at least two of Gradient Descent, Stochastic Gradient Descent (SGD), Mini-batch Gradient Descent, Nesterov Accelerated Gradient (NAG), AdaGrad, RMSProp, Adam (Adaptive Moment Estimation), AdaDelta, Nadam (Nesterov-accelerated Adaptive Moment Estimation), Learning Rate Decay, Early Stopping, Weight Decay, Dropout, and Batch Normalization.

[0127] The fluorescent signal point recognition method provided by the above-mentioned embodiment of the present disclosure performs image feature extraction on a multi-channel fluorescent signal point image to be identified including the same cell object based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; then performs multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; finally, based on the fluorescent signal point recognition model, performs fluorescent signal point recognition on the multi-scale fusion feature map to obtain the fluorescent signal point recognition result of the fluorescent signal point image to be identified in each channel. That is, only the fluorescent signal point images of the multi-channels taken under the same field of view need to be input, and the fluorescent signal point recognition result of the fluorescent signal point image of each channel can be output. Thus, end-to-end fluorescent signal point recognition can be realized, and fluorescent signal point recognition can be realized based on the multi-channel fluorescent signal point images, and the fluorescent signal point recognition can be completed with fewer model parameters, which saves the amount of calculation and improves the calculation efficiency.

[0128] Embodiment 1:

[0129] In the specific experiment, the experimental data came from 82 samples, and 1 to 4 groups of four-channel (specifically including) fluorescent signal point images were extracted from each sample. The selection criterion was that the cell density in each fluorescent signal point image was relatively uniform. Finally, 110 groups of multi-channel fluorescent signal point images were formed, and 20 labelers annotated the fluorescent signal point recognition results for each fluorescent signal point image, and 5 senior labelers reviewed them. The above 110 samples were divided into training sets and validation sets. The training set is 60 groups of sample data (specifically including multi-channel fluorescent signal point images and corresponding annotated fluorescent signal point recognition results), and the test set is 50 groups of sample data.

[0130] The model training process is as follows:

[0131] 1) Preprocess the input image:

[0132] According to the cell contour annotation of the FISH image, the fluorescent signal point images of the four channels (Red, Green, Gold and Aqua) are cut into cell-based signal point images and converted into grayscale images. The signal point images of the four channels are then merged into one image with a dimension of (320, 320, 4).

[0133] 2) Data enhancement:

[0134] The following data augmentation is performed on the grayscale image after merging the four channels: random brightness enhancement, letterbox enhancement, HSV color enhancement, random perspective enhancement, grayscale and normalization.

[0135] 3) Training methods:

[0136] The data after the above enhancement processing is input into Figure 3 The preliminary fusion feature extraction model, multi-scale feature extraction model and fluorescent signal point recognition model shown in the figure, calculate the loss value between the output of the model and the labeled fluorescent signal point recognition result according to the loss function, use Adam as the optimizer, the learning rate is 1.0e-3, weight_decay is 5e-4, and update the parameters of the preliminary fusion feature extraction model, multi-scale fusion extraction model and fluorescent signal point recognition model. Use ReduceLROnPlateau as lr_scheduler, factor is 0.9, patience is 5, and adjust the learning rate of the optimizer.

[0137] Train for 100 epochs and use F1 score as the evaluation index. The maximum sum of F1 scores of the four-channel signal point images is taken as the best model. The algorithm uses 0.3 as the threshold.

[0138] Loss function: The loss function of the signal point heat map extraction model uses focal loss, and the loss functions of the signal point width extraction model, signal point height extraction model, signal point position horizontal coordinate offset extraction model, and signal point position vertical coordinate offset extraction model all use reg_l1_loss.

[0139] The model inference process is as follows:

[0140] The cell-based four-channel fluorescence signal point images are merged into one image with a dimension of (1, 4, 320, 320), which is input into the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescence signal point recognition model to obtain the position and size of the signal points of the cell-based four-channel fluorescence signal point image.

[0141] Please refer to Table 1 for specific experimental results:

[0142] Table 1. Performance comparison of algorithms

[0143]

[0144] In Table 1, Fnet corresponds to the experimental data of the fluorescent signal point recognition method disclosed in Chinese patent application 202210958372X, and FnetV2 is the experimental data of the specific experiment of the present disclosure.

[0145] The FLOPS in Table 1 is the result obtained by using two methods to identify fluorescent signal points on a randomly generated test data.

[0146] The prediction time in Table 1 is the mean prediction time for all data in all test sets.

[0147] In Chinese patent application 202210958372X, it is necessary to first complete signal point detection for the signal points of the fluorescent signal point image of each channel, and then complete joint training for the four-channel fluorescent signal point images, so as to complete the signal point detection of the four-channel signal point images at the same time. The FnetV2 algorithm adopted in the present disclosure can read the four-channel signal point images at the same time during the training process, perform feature extraction at the same time, and complete the signal point detection of the four-channel signal point images at the same time. As can be seen from Table 1, compared with Fnet, FnetV2 reduces the storage space required for model parameters, reduces the amount of calculation, and thus improves the calculation speed.

[0148] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a fluorescent signal point recognition device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0149] like Figure 6 As shown, the fluorescent signal point recognition device 600 of this embodiment includes: an acquisition module 601, a feature preliminary fusion module 602, a multi-scale feature extraction module 603 and a fluorescent signal point recognition module 604. The acquisition module 601 is configured to acquire a multi-channel fluorescent signal point image to be recognized obtained by shooting the same field of view including the same cell object; the feature preliminary fusion module 602 is configured to perform image feature extraction on the multi-channel fluorescent signal point image to be recognized based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; the multi-scale feature extraction module 603 is configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; the fluorescent signal point recognition module 604 is configured to perform fluorescent signal point recognition on the multi-scale fusion feature map based on the fluorescent signal point recognition model to obtain a fluorescent signal point recognition result of the fluorescent signal point image to be recognized in each channel.

[0150] In this embodiment, the specific processing of the acquisition module 601, the feature preliminary fusion module 602, the multi-scale feature extraction module 603 and the fluorescent signal point recognition module 604 of the fluorescent signal point recognition device 600 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment are not repeated here.

[0151] In some optional embodiments, the multi-scale feature extraction model includes a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an upsampling network, a splicing network and a fourth C2f network connected sequentially, and the outputs of the second C2f network and the upsampling network are input into the splicing network.

[0152] In some optional embodiments, the fluorescent signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and

[0153] The fluorescent signal point recognition module is further configured as follows:

[0154] Based on the signal point area information map extraction model, the signal point area information map is extracted from the multi-scale fusion feature map to obtain a signal point area information map of the fluorescence signal point image to be identified in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescence signal point image;

[0155] Based on the signal point heat map extraction model, the signal point heat map is extracted from the multi-scale fusion feature map to obtain a signal point heat map of the fluorescent signal point image to be identified in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map;

[0156] For each channel of the fluorescent signal point image to be identified, the fluorescent signal point identification result of the fluorescent signal point image to be identified in the channel is determined based on the signal point area information map and the signal point heat map of the fluorescent signal point image to be identified in the channel.

[0157] In some optional embodiments, the signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer, and an activation function layer connected in sequence.

[0158] In some optional embodiments, the first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

[0159] In some optional implementations, the signal point area information graph extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model, and a signal point position vertical coordinate offset extraction model.

[0160] In some optional embodiments, the signal point width extraction model includes a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model includes a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model includes a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model includes a sixth image feature extraction network and a convolutional layer connected in sequence.

[0161] In some optional embodiments, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are pre-trained through the following training steps:

[0162] Acquire a sample data set, wherein the sample data includes a multi-channel sample fluorescence signal point image and annotated fluorescence signal point recognition results of the sample fluorescence signal point image of each channel, wherein the multi-channel sample fluorescence signal point image is a multi-channel fluorescence signal point image obtained by photographing the same field of view including the same cell object;

[0163] Based on the sample data set, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are trained and obtained.

[0164] It should be noted that the implementation details and technical effects of each module in the fluorescent signal point identification device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0165] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 suitable for implementing the electronic device of the present disclosure. Figure 7 The computer system 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0166] like Figure 7 As shown, the computer system 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. Various programs and data required for the operation of the computer system 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0167] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the computer system 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The computer system 700 of the electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0168] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0169] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0170] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0171] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2 The illustrated embodiment and its optional implementations illustrate a method for identifying fluorescent signal points.

[0172] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Python, Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0173] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the acquisition module may also be described as "a module for acquiring a multi-channel image of fluorescent signal points to be identified obtained by photographing the same field of view including the same cell object".

[0175] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A method for identifying fluorescent signal points, comprising: Acquire a multi-channel image of fluorescent signal points to be identified by photographing the same field of view including the same cell object; Based on the preliminary fusion feature extraction model, image feature extraction is performed on the multi-channel fluorescent signal point image to be identified to obtain a preliminary fusion feature map; Based on a multi-scale feature extraction model, multi-scale feature extraction is performed on the preliminary fusion feature map to obtain a multi-scale fusion feature map; Based on the fluorescence signal point recognition model, fluorescence signal point recognition is performed on the multi-scale fusion feature map to obtain the fluorescence signal point recognition result of the fluorescence signal point image to be recognized in each channel.

2. The method according to claim 1, wherein: The multi-scale feature extraction model includes a first C2f network, a first convolutional neural network, a second C2f network, a second convolutional neural network, a third C2f network, an SPPF network, an upsampling network, a splicing network and a fourth C2f network connected in sequence, and the outputs of the second C2f network and the upsampling network are input into the splicing network.

3. The method according to claim 1, wherein: The fluorescent signal point recognition result includes a signal point area information map and a signal point heat map of a preset feature map size, and the fluorescent signal point recognition model includes a signal point area information map extraction model and a signal point heat map extraction model; and The method of performing fluorescence signal point recognition on the multi-scale fusion feature map based on the fluorescence signal point recognition model to obtain the fluorescence signal point recognition result of the fluorescence signal point image to be recognized in each channel includes: Based on the signal point area information map extraction model, the signal point area information map is extracted from the multi-scale fusion feature map to obtain a signal point area information map of the fluorescence signal point image to be identified in each channel, wherein the signal point area information map is used to map each pixel point in the feature map under the preset feature map size to an image area in the fluorescence signal point image; Based on the signal point heat map extraction model, the signal point heat map is extracted from the multi-scale fusion feature map to obtain a signal point heat map of the fluorescent signal point image to be identified in each channel, wherein the signal point heat map includes the probability that each pixel point in the feature map under the preset feature map size is a signal point area after being mapped to the image area in the fluorescent signal point image according to the signal point area information map; For each channel of the fluorescent signal point image to be identified, the fluorescent signal point identification result of the fluorescent signal point image to be identified in the channel is determined based on the signal point area information map and the signal point heat map of the fluorescent signal point image to be identified in the channel.

4. The method according to claim 3, wherein: The signal point heat map extraction model includes a first image feature extraction network, a second image feature extraction network, a convolution layer and an activation function layer which are connected in sequence.

5. The method according to claim 4, wherein: The first image feature extraction network and the second image feature extraction network respectively include a convolution layer, a batch normalization layer and an activation function layer connected in sequence.

6. The method according to claim 3, wherein: The signal point area information graph extraction model includes a signal point width extraction model, a signal point height extraction model, a signal point position horizontal coordinate offset extraction model and a signal point position vertical coordinate offset extraction model.

7. The method according to claim 6, wherein: The signal point width extraction model includes a third image feature extraction network and a convolutional layer connected in sequence, the signal point height extraction model includes a fourth image feature extraction network and a convolutional layer connected in sequence, the signal point position horizontal coordinate offset extraction model includes a fifth image feature extraction network and a convolutional layer connected in sequence, and the signal point position vertical coordinate offset extraction model includes a sixth image feature extraction network and a convolutional layer connected in sequence.

8. The method according to claim 1, wherein: The preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are pre-trained through the following training steps: Acquire a sample data set, wherein the sample data includes a multi-channel sample fluorescence signal point image and annotated fluorescence signal point recognition results of the sample fluorescence signal point image of each channel, wherein the multi-channel sample fluorescence signal point image is a multi-channel fluorescence signal point image obtained by photographing the same field of view including the same cell object; Based on the sample data set, the preliminary fusion feature extraction model, the multi-scale feature extraction model and the fluorescent signal point recognition model are trained and obtained.

9. A fluorescent signal point recognition device, comprising: An acquisition module is configured to acquire a multi-channel image of fluorescent signal points to be identified obtained by photographing a same field of view including a same cell object; A feature preliminary fusion module is configured to extract image features from the multi-channel fluorescent signal point image to be identified based on a preliminary fusion feature extraction model to obtain a preliminary fusion feature map; A multi-scale feature extraction module is configured to perform multi-scale feature extraction on the preliminary fusion feature map based on a multi-scale feature extraction model to obtain a multi-scale fusion feature map; The fluorescence signal point recognition module is configured to perform fluorescence signal point recognition on the multi-scale fusion feature map based on the fluorescence signal point recognition model to obtain the fluorescence signal point recognition result of the fluorescence signal point image to be recognized in each channel.

10. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images