A target molecule observation method based on human-computer interaction

Abstract

The application discloses a target molecule observation method based on human-computer interaction. The method first uses a deep neural network to process input pictures to obtain deep features of the images; a user uses a mouse to interact with the system, and the system can understand the intention; then, the target features selected by the user are used as a target template to perform similarity matching in the deep feature layer, and the similar regions are mapped back to the original image to obtain similar targets; for dynamic images, the obtained similar targets can be tracked to observe dynamic changes. The advantage of the application is that the detection of specific substances does not need to be specially trained, that is, a model trained only for A data set can also have good detection effect on the detection of other targets such as B, C and D; in addition, after the similar targets are detected, the targets can be tracked to obtain detailed dynamic change information of the substances in a period of time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a target molecule observation method based on human-computer interaction. BACKGROUND

[0002] From a microscopic perspective, most life activities in nature are completed by a large number of nucleic acids, proteins and other molecules. Observing molecules is an important means to find out the role played by cells, bacteria, viruses and the like in life activities. For example, in the process of dairy fermentation, it is necessary to monitor the types and population sizes of bacterial colonies to ensure the quality of dairy fermentation; when studying the pathogenic mechanism of the new coronavirus, it is necessary to observe the interaction between key proteins and host cells. The information obtained by observing target molecules, such as the number of molecules, the degree of aggregation, the degree of activity and the interaction, is an important basis for revealing the internal mechanism of life activities in the field of life sciences. In order to obtain this information, researchers often need to manually identify, label and distinguish the changes of a large number of molecules. This is a heavy and tedious work, and in addition, due to the different levels and experiences of researchers, there is a large subjective nature in recording the activities of molecules. Therefore, it is urgent to develop a method capable of quickly and accurately detecting target molecules and continuously tracking target molecules.

[0003] Early researches mainly aimed at the detection of cells and micro substances, and the main methods included threshold segmentation based on morphological operation, region-based and edge-based K-Means clustering algorithm and the like. These traditional methods have relatively simple algorithms and cannot consider the deep features of cell images, so the detection accuracy is not high in complex situations. In addition, there are other methods based on artificial design of specific cell image features and machine learning classifiers to recognize and detect. These methods are applied more in early cell detection, but they cannot utilize the prior information of cells, cannot effectively solve the segmentation of overlapping cells and adherent cells, and are not efficient. They can only recognize and detect specific cells and specific features, and have some limitations. They cannot fully utilize the original image, and many deep features are lost. In recent years, image segmentation methods based on convolutional neural networks, such as FCN, U-Net, SegNet and the like, have greatly improved the accuracy compared with traditional algorithms. Compared with traditional algorithms, the convolutional neural network algorithm can abstract data to a higher level through multiple nonlinear transformations without artificial feature design for specific scenarios. In addition, in the process of image segmentation, the convolutional neural network enhances the robustness of feature extraction through local connection and weight sharing, and is increasingly applied to various fields of medical image analysis, such as cell counting, cell segmentation and cell detection.

[0004] But the existing various technologies can not well meet the actual demand, the following problems have not been effectively solved: 1) the target detection technology can only effectively detect the trained target, and cannot accurately identify the same kind for the untrained target; 2) there are multiple same category targets in part of the image, researchers hope to select some interesting targets for attention, and the existing technology cannot well interact with researchers; 3) multiple substances need to be observed simultaneously during the life activity, and the existing research is only effective for detecting one substance; 4) during the long-time dynamic movement of various substances, it is impossible to continuously track the various data information of the target. SUMMARY

[0005] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0006] In view of the above existing problems, the present application is proposed.

[0007] Therefore, the technical problems solved by the present application are: 1) the existing target detection technology can only effectively detect the trained target, and cannot accurately identify the same kind for the untrained target; 2) there are multiple same category targets in part of the image, researchers hope to select some interesting targets for attention, and the existing technology cannot well interact with researchers; 3) multiple substances need to be observed simultaneously during the life activity, and the existing research is only effective for detecting one substance; 4) during the long-time dynamic movement of various substances, it is impossible to continuously track the various data information of the target.

[0008] To solve the above technical problems, the present application provides the following technical scheme: a target molecule observation method based on human-computer interaction, comprising:

[0009] The trained U-Net deep neural network is used to process the input image I, and the result matrix obtained by intercepting a certain up-sampling layer is taken as a deep feature vector F;

[0010] The user frames the target of interest in the image I, and finds the corresponding feature block in the feature vector F of the image as a template;

[0011] The similarity of each part is calculated in the feature vector F of the whole graph by sliding window, and if the similarity is greater than a set threshold, it is considered as the same kind of target;

[0012] The coordinates of the same type of target are converted from the feature space to the coordinates on the image, and the method of non-maximum suppression is used to remove the overlapping frame, to obtain the same type of target;

[0013] In the dynamic image, the obtained same type of target is dynamically tracked.

[0014] As a preferred scheme of the target molecule observation method based on human-computer interaction provided by the application, the training method of the U-Net deep neural network comprises:

[0015] A specific data set is used as a training set, the set includes target original images and labeled target mask images, and the set is used for training and verification of the model according to 8:2; the Adam algorithm is selected for end-to-end training, the learning rate is set to 0.001, the binary cross entropy is used as the loss function, the Batch-size is 8, the Epoch is 50, and the maximum Dice coefficient obtained by the verification set is used as the final model.

[0016] As a preferred scheme of the target molecule observation method based on human-computer interaction provided by the application, the size of the image I is n*n, and the size of the deep feature vector F is c*m*m, wherein c is the channel number of the feature matrix, and m is the width and height of the feature matrix.

[0017] As a preferred scheme of the target molecule observation method based on human-computer interaction provided by the application, the user frames the target of interest in the original image, comprising:

[0018] The user uses a mouse to frame the target of interest to obtain IB, and records the upper left corner coordinates (i, j) and the width and height (w0, h0) of the frame.

[0019] As a preferred scheme of the target molecule observation method based on human-computer interaction provided by the application, the corresponding feature block in the feature vector of the image is found as a template, comprising:

[0020] The framed target molecule is mapped to the feature map F to obtain the feature matrix FB of the target molecule, and according to the proportional relationship between the picture I and the feature map F, the coordinates of the upper left corner of FB in the feature map are (p, q), the width and height are (w F ,h F ), and are represented as follows:

[0021]

[0022] As a preferred scheme of the target molecule observation method based on human-computer interaction provided by the application, wherein:

[0023] The similarity of each part is calculated in the feature vectors of the whole graph by means of the sliding window, including:

[0024] The feature matrix FB of the target molecule is matched in the feature matrix F of the whole molecular image in the order from left to right and from top to bottom, and the step length of horizontal and vertical movement is s h and s v , wherein s h ∈(0,4], s v ∈(0,4], s h and s v are integers;

[0025] Suppose the feature block corresponding to the feature matrix FB of the target molecule in the feature matrix F is FB k , wherein k represents the number of movements, and suppose the upper left corner coordinates of FB k are (p k , q k ), which are represented as follows:

[0026] p k =s h ×s v ×k / (n-s v )

[0027] q k =s v ×k mod(n-s v / s v )

[0028] As a preferred scheme of the target molecule observation method based on human-computer interaction, the similarity of each part includes:

[0029] First, normalization processing is performed, and the following is calculated:

[0030] and

[0031] Wherein ||FB|| and ||FB k || represent the norm of FB and FB k , respectively, and then the similarity Similarity of the one-dimensional vectors FB' and FB k ' is calculated by using the cosine distance, and the calculation is as follows:

[0032]

[0033] If the similarity is greater than the threshold value, the feature block corresponds to a molecule of the same type as the target molecule, otherwise move to the next position for calculation until the entire feature layer is traversed, wherein the threshold value can be adjusted according to the needs of the user, and the threshold value is in the range of (0, 1).

[0034] As a preferred scheme of the target molecule observation method based on human-computer interaction, the method comprises the following steps:

[0035] The feature block corresponding to the molecule of the same type as the target molecule is mapped to the molecular image to form a candidate region, the upper left corner coordinates of the candidate region are (i k ,j k ), the width and height are (w F ,h F ), and the candidate region is represented as follows:

[0036]

[0037] In order to ensure that all molecules of the same type can be accurately detected, the step length of horizontal and vertical movement of the feature matrix FB of the target molecule is s h and s v respectively when performing the movement matching, wherein s h ∈(0, 4], s v ∈(0, 4], s h and s v are all integers; this makes each molecule of the same type be covered by multiple candidate regions.

[0038] As a preferred scheme of the target molecule observation method based on human-computer interaction, the method comprises the following steps:

[0039] The candidate region most similar to the feature matrix FB of the target molecule is selected from the multiple candidate regions as a molecule of the same type as the target molecule by using the non-maximum suppression method, so that the final multiple accurate targets are obtained.

[0040] As a preferred scheme of the target molecule observation method based on human-computer interaction, the method comprises the following steps:

[0041] After the image of the first frame is processed by the above steps, the molecule of the same type is detected, and then the CSR-DCF tracking algorithm is used to track the target in each frame of image.

[0042] The present application has the following beneficial effects:

[0043] (1) Compared with the prior art, the target detection technology of the present application can not only effectively detect the trained category of targets, but also has good detection effect for data that has not been trained.

[0044] (2) The present application has human-computer interaction capability, can understand the intention of the user, and help the user to complete the observation of the target of interest and the like.

[0045] (3) The present application can distinguish different targets of different categories in a picture, and simultaneously observe different targets of the same category.

[0046] (4) The present application can track the dynamic process of the target in the life activity for a long time, and ensure continuous observation. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0048] Figure 1 A basic flowchart of a target molecule observation method based on human-computer interaction provided for an embodiment of the present application;

[0049] Figure 2 A training set original picture and a template example of a target molecule observation method based on human-computer interaction provided for the first embodiment of the present application;

[0050] Figure 3 A structure diagram of a U-Net neural network of a target molecule observation method based on human-computer interaction provided for the first embodiment of the present application;

[0051] Figure 4 A structure diagram of a U-Net neural network with attention mechanism of a target molecule observation method based on human-computer interaction provided for the second embodiment of the present application;

[0052] Figure 5 A to-be-detected picture of a target molecule observation method based on human-computer interaction provided for the second embodiment of the present application;

[0053] Figure 6 A schematic diagram of a user framing the target of interest in the picture to be detected of a target molecule observation method based on human-computer interaction provided for the second embodiment of the present application;

[0054] Figure 7An experimental detection result graph of a target molecule observation method based on human-computer interaction provided by the second embodiment of the present application;

[0055] Figure 8 An experimental detection result graph of a target molecule observation method based on human-computer interaction provided by the second embodiment of the present application;

[0056] Figure 9 A performance curve graph of different target molecule detection networks on a cell nucleus data set for a target molecule observation method based on human-computer interaction provided by the third embodiment of the present application;

[0057] Figure 10 A performance curve graph of different target molecule detection networks on a human protein atlas data set for a target molecule observation method based on human-computer interaction provided by the third embodiment of the present application;

[0058] Figure 11 A performance curve graph of different target molecule detection networks on a bacteria data set for a target molecule observation method based on human-computer interaction provided by the third embodiment of the present application;

[0059] Figure 12 A performance curve graph of a target molecule detection network on a red blood cell data set for a target molecule observation method based on human-computer interaction provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present application.

[0061] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0062] Secondly, the "one embodiment" or "embodiment" referred to herein can include specific features, structures or characteristics contained in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0063] The application is described in detail in conjunction with the schematic diagram. In the detailed description of the embodiments of the application, the cross-sectional view of the device structure is partially enlarged without the general proportion for the convenience of illustration, and the schematic diagram is only an example, which should not limit the scope of protection of the application herein. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual production.

[0064] Meanwhile, in the description of the application, it should be noted that the terms "upper, lower, inner and outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application. In addition, the terms "first, second or third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0065] Unless otherwise specified and limited, the terms "mounting, connecting, connecting" in the application should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; it can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0066] Embodiment 1

[0067] Reference Figures 1-3 For an embodiment of the application, a target molecule observation method based on human-computer interaction is provided, comprising:

[0068] S1: processing the input image I by using the trained U-Net deep neural network, and taking the result matrix obtained by intercepting a certain up-sampling layer as a deep feature vector F;

[0069] Further, training the U-Net deep neural network comprises:

[0070] A1: taking the Nucleus data set as the training set, as shown in Figure 2 , which shows the original image and template example of the training set, the set includes target original image and labeled target mask image, which is used for training and verification of the model according to 8:2;

[0071] It should be noted that the dataset contains labeled hematoxylin-eosin staining (H&E) images, which are one of the most common types of images in histopathology; these images are cropped from digitalized tissue samples in a whole-slide imaging system, which are from 7 organs mentioned in Cancer and Tumor Atlas in the Genome (CATIAG) project. The dataset integrates 30 cropped images, containing more than 21,000 nuclei; and each image is labeled and verified by medical experts, which can be used by researchers to develop and test more universal nuclear segmentation techniques for various types of nuclei.

[0072] A2: Train the U-Net convolutional neural network with the sample training set in step A1, as shown in Figure 3 , which shows the structure of the U-Net convolutional neural network, which is implemented based on PyTorch, specifically using the Adam algorithm for end-to-end training with a learning rate of 0.001, using binary cross-entropy as the loss function, with a batch size of 8 and an epoch of 50, and taking the maximum Dice coefficient on the validation set as the final model.

[0073] It should be noted that the U-Net deep neural network is an image segmentation method based on convolutional neural networks that enhances the robustness of feature extraction through local connections, weight sharing, and other techniques, and is increasingly used in various fields of medical image analysis, such as cell counting, cell segmentation, and cell detection.

[0074] The Dice coefficient is a measure of the similarity of sets, commonly used to measure the performance of biomedical image segmentation. The Dice coefficient is calculated as follows:

[0075]

[0076] where Pred is the segmentation map and Seg is the actual segmentation map. Dice ∈ [0, 1], when the segmentation map is completely consistent with the actual segmentation map, Dice = 1, and the worst Dice = 0.

[0077] Further, extracting the image deep feature vector includes:

[0078] A3: Use the trained U-Net convolutional neural network in step A2 to process the image and extract the image deep feature vector:

[0079] Specifically, input the image I to be processed with a size of n x n into the trained U-Net neural network for processing, obtain the result of a certain upsampling layer, and get a matrix with a size of c x m x m as the feature F, where c is the number of channels of the feature matrix, and m is the width and height of the feature matrix.

[0080] It should be noted that the image size n affects the processing efficiency; the larger n is, the larger the computational scale, but the better the effect. The number of channels c, width, and height m of the matrix affect the accuracy of the results and need to be adjusted according to the network model structure used.

[0081] S2: Select the target of interest in image I, and find its corresponding feature block in the feature vector of the image as a template;

[0082] Furthermore, the user uses the mouse to select the target of interest, obtains the IB, and records the coordinates of its upper left corner as (i,j), and the width and height of the box as (w0,h0);

[0083] Furthermore, by mapping the selected target molecules onto feature map F, the feature matrix of the target molecules is obtained as FB. Based on the ratio between image I and feature map F, the coordinates of the upper left corner of FB in the feature map are (p, q), and the width and height are (w, q). F ,h F The ratio between image I and feature map F is as follows:

[0084]

[0085] S3: Calculate the similarity of each part in the feature vector of the entire image one by one by using a sliding window. If the similarity is greater than the set threshold, it is considered to be the same type of target.

[0086] Furthermore, the sliding window method includes:

[0087] Using the feature matrix FB of the target molecule, sliding matching is performed in the feature matrix F of the entire molecular image from left to right and from top to bottom, with step sizes s in the horizontal and vertical directions, respectively. h and s v , where s h ∈(0,4],s v ∈(0,4],s h and s v All are integers;

[0088] Let FB be the feature block in the feature matrix F that corresponds to the feature matrix FB of the target molecule. k , set FB k The coordinates of the top left corner are (p k ,q k ), which is represented as follows:

[0089] p k =s h ×s v ×k / (ns v )

[0090] q k= s v x k mod (n-s v / s v )

[0091] wherein k represents the number of movements.

[0092] Further, the cosine similarity between the target molecule feature and the feature block is calculated, comprising:

[0093] B1: normalizing, calculating as follows:

[0094] and

[0095] wherein ||FB|| and ||FB k || represent the norm of FB and FB k , respectively, and then the similarity Similarity between FB' and FB k ' one-dimensional vectors is calculated using the cosine distance, calculating as follows:

[0096]

[0097] wherein FB' T represents the transpose of the matrix FB'.

[0098] B2: if the similarity Similarity is greater than a threshold value θ, θ∈(0, 1), the feature block corresponds to a molecule of the same class as the target molecule, otherwise move to the next position for calculation until the entire feature layer is traversed.

[0099] It should be noted that the threshold value θ can be visually adjusted according to the needs of the user.

[0100] S4: the coordinates of the same class target are converted from the feature space back to the coordinates on the image, and the non-maximum suppression method is used to remove overlapping boxes to obtain the same class target:

[0101] Further, the feature block corresponding to the molecule of the same class as the target molecule is mapped to the molecular image to form a candidate region, the upper left corner coordinates are (i k ,j k ), the width and height are (w F ,h F ), and are represented as follows:

[0102]

[0103] In order to ensure that all molecules of the same class can be accurately detected, the feature matrix FB of the target molecule will be covered by multiple candidate regions when performing movement matching.

[0104] Further, a non-maximum suppression method is used to select a candidate region most similar to the target molecule feature matrix FB from multiple candidate regions as a molecule of the same type as the target molecule, thereby obtaining a plurality of accurate targets.

[0105] It should be noted that non-maximum suppression in target detection is to suppress elements that are not maximum values, which can be understood as local maximum search, and is a method of extracting the window with the highest score.

[0106] S5: dynamically tracking the same target selected in S4:

[0107] Further, the above steps are performed on the first frame image to obtain all targets to be observed.

[0108] Further, the CSR-DCF tracking algorithm is used to track the target in each frame of image to realize synchronous tracking of all targets.

[0109] It should be noted that CSR-DCF is an image segmentation method using spatial reliability and channel reliability, which can more accurately select an effective tracking target area and has been widely applied in the field of multi-target tracking technology.

[0110] The method described in the embodiment is used to detect the same target in different data sets, and the test results are shown in Table 1.

[0111] Table 1: Mean value of experimental data of U-Net on each data set

[0112]

[0113] The Nucleus data set is a training data set, and the average accuracy is 0.868; the HPA, Bacteria and Red bloodcell are data sets that have not been trained, and the average accuracies are 0.877, 0.9783 and 0.8694, respectively, which shows that the method also has high performance on data sets that have not been trained. It can well meet the detection needs of the life process changes of micro-substances such as substances or cells in life science research.

[0114] Embodiment 2

[0115] Reference Figures 4-9 For an embodiment of the present application, a target molecule observation method based on human-computer interaction is provided, and scientific demonstration is carried out through actual application to verify the beneficial effects of the present application.

[0116] An attention mechanism is added to the up-sampling layer of U-Net (referred to as Attention-based U-Net), see Figure 4This demonstrates the structure after adding an attention mechanism to the upsampling layer of U-Net;

[0117] The U-Net convolutional neural network was trained using the Nucleus dataset as the training set. The Adam algorithm was selected for end-to-end training with a learning rate of 0.001, a binary cross-entropy loss function, a batch size of 8, and an epoch of 50. The model with the largest Dice coefficient obtained from the validation set was used as the final model.

[0118] visible Figure 5 For the image to be detected, the image I with a size of 512×512 is input into the trained U-Net neural network for processing, and the resulting matrix of size 32×128×128 is used as the feature F.

[0119] visible Figure 6 The example demonstrates how a user selects a target of interest using the mouse, obtains the IB (Image I), and records the coordinates of its top-left corner as (178, 134), and the width and height of the box as (42, 36). The selected target molecule is then mapped onto the feature map F to obtain the feature matrix of the target molecule as FB. Based on the ratio between image I and feature map F, the coordinates of the top-left corner of FB in the feature map are (44, 33), and the width and height are (10, 9).

[0120] Using the feature matrix FB of the target molecule, sliding matching is performed in the feature matrix F of the entire molecular image from left to right and from top to bottom, with step sizes s in the horizontal and vertical directions, respectively. h =4 and s v =4;

[0121] The threshold θ is set to 0.75. Let FB be the feature block in the feature matrix F that corresponds to the feature matrix FB of the target molecule. k FB k The coordinates of the top left corner are (p k ,q k ), calculate the cosine similarity and one-dimensional vector similarity of the target molecule features and feature blocks. If there is a certain FB k If the Similarity is greater than θ, then the feature blocks corresponding to molecules of the same type as the target molecule are mapped onto the molecular image to form candidate regions, with the upper left corner coordinates of the region being (i... k ,j k ), with a width and height of (10,9).

[0122] It should be noted that more than 15,000 detection boxes will be generated here based on the movement of the sliding window, and each box will have a similarity coordinate (i, j).

[0123] The method of non-maximum suppression is used to select the candidate region most similar to the target molecular feature matrix FB from multiple candidate regions to obtain the final multiple accurate targets, and it can be seen from Figure 7 , which shows the detection results in the experiment; the selected similar targets are tracked using the CSR-DCF tracking algorithm for each target in the frame, and it can be seen from Figure 8 , which shows the tracking results of the target motion in the dynamic situation.

[0124] The method described in the embodiment is used to detect similar targets in different data sets, and the test results are shown in Table 2.

[0125] Table 2: Mean value of experimental data of Attention-based U-Net on each data set

[0126]

[0127] Embodiment 3

[0128] Reference Figures 9-12 For another embodiment of the application, which is different from the first two embodiments, in order to verify the technical effects used in the method, the embodiment compares the test results of the traditional technical solution and the method of the application by scientific demonstration means, in order to verify the real effect of the method.

[0129] In order to verify the effectiveness of U-Net convolutional neural network and attention-based U-Net neural network in detecting similar molecules to target molecules, the embodiment is compared with FCN32s network, SegNet network and CE-Net network. These neural network models are realized based on PyTorch, and Adam algorithm is selected for end-to-end training, the learning rate is set to 0.001, binary cross entropy is used as the loss function, Batch-size is 8, and Epoch is 50, and the maximum Dice coefficient obtained from the validation set is used as the final model.

[0130] It can be seen from Figures 9-12 , which shows the accuracy, recall rate and PR curve value of different target molecule detection networks on the test set of the cell nucleus data set, human protein atlas data set, bacterial data set and red blood cell data set, and it should be noted that (a) is the accuracy curve; (b) is the recall rate curve; (c) is the PR curve, and the area under the PR curve is the average precision AP.

[0131] As can be seen from the figure, with the increase of the similarity threshold, the accuracy of the five target molecule detection networks shows an upward trend and gradually approaches 1. The average accuracy of different target molecule detection networks on each data set is shown in Table 3.

[0132] Table 3 Average precision of different target molecule detection networks on each dataset

[0133]

[0134]

[0135] As can be seen from Table 3, on the test set of the nucleus data, the deformed form of U-Net, Attention-based U-Net, has the highest average precision, reaching 0.9586, which can well meet the needs of the interactive target molecule detection task. The average precision of FCN32s reaches 0.9026, which is inferior to Attention-based U-Net, but by Figure 9 It can be seen that its recall rate is the worst among all networks. The average precision of U-Net is 0.8679, ranking third. The average precision of SegNet is 0.6234, which has the worst performance. From Figures 10-12 and Table 3, on the three datasets of human protein atlas, bacteria and red blood cells that did not participate in training, the precision, recall rate and average precision of SegNet also performed poorly. Attention-based U-Net performed best on the human protein atlas and bacteria datasets, while U-Net performed better than Attention-based U-Net on the red blood cell dataset. This shows that the method in this paper can also achieve good performance on untrained molecular images, and can meet the actual needs of scientific research in the field of life sciences.

[0136] It should be noted that, as can be seen from Figure 12 , the performance indicators of different target molecule detection networks are significantly different from Figure 10 and 11 The main reason is that in the red blood cell image, there are many red blood cells, and there are a large number of overlaps, making it difficult to detect cells of the same kind. On the red blood cell dataset, the proposed U-Net and Attention-based U-Net still achieved the highest average precision value.

[0137] Table 4 is the average value of the average precision of different target molecule detection networks in the experiment, Attention-based U-Net reaches the highest average precision value of 0.9125, and U-Net reaches 0.8981, which is only second to Attention-based U-Net, which also shows that adding an attention module helps to improve the stability of the method.

[0138] Table 4 Average precision of different target molecule detection networks

[0139]

[0140] In summary, the U-Net and Attention-based U-Net proposed in this paper can still achieve the detection of target molecules for molecular images that do not participate in training, and the performance is better than other networks, and the stability of detection can be improved by increasing the attention module.

[0141] It should be appreciated that embodiments of the present application can be realized or implemented by computer hardware, a combination of hardware and software, or through computer instructions stored in a non-transitory computer-readable memory. The method can be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium thus configured causes a computer to operate in a specific and predefined manner according to the methods described in the specific embodiments and the accompanying drawings. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed application-specific integrated circuit for this purpose.

[0142] In addition, the operations of the processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions (e.g., computer programs, one or more computer programs, or one or more applications) to perform operations, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. The computer programs include a plurality of instructions executable by one or more processors.

[0143] Further, the methods can be implemented in any type of computing platform operatively coupled to a suitable computing platform, including but not limited to a personal computer, a mini-computer, a mainframe, a workstation, a network or distributed computing environment, a stand-alone or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the present application can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated to the computing platform, such as a hard disk, an optical read and / or write storage medium, RAM, ROM, and the like, such that it can be read by a programmable computer to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. In addition, the machine-readable code, or portions thereof, can be transmitted over a wired or wireless network. The present application, as described herein, includes these and other non-transitory computer-readable storage media of different types when such media include instructions or programs implementing the steps described above in connection with a microprocessor or other data processor. The present application also includes the computer itself when programmed according to the methods and techniques described herein. The computer program can be applied to input data to perform the functions described herein and thereby transform the input data into output data that can be stored to non-volatile memory. The output information can further be applied to one or more output devices, such as a display. In a preferred embodiment of the present application, the transformed data represents a physical and tangible object, including a particular visual depiction of a physical and tangible object produced on a display.

[0144] As used in this application, the terms "component," "module," "system" and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution and a component can be localized, partially and / or entirely, in one computer or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as the Internet with other systems via the signal).

[0145] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A method for observing a target molecule based on human-machine interaction, characterized by, The method comprises the following steps: An input image I is processed by using a trained U-Net deep neural network, and a result matrix obtained by upsampling a certain layer of the U-Net deep neural network is taken as a deep feature matrix F; The training method of the U-Net deep neural network comprises the following steps: a specific target molecule data set is taken as a training set, the training set comprises a target original image and an image with a labeled target mask, the training set is divided into two parts according to a ratio of 8:2 for training and verification of the model, an Adam algorithm is selected for end-to-end training, a learning rate is set to 0.001, a binary cross-entropy is taken as a loss function, a batch-size is 8, an epoch is 50, and a maximum Dice coefficient obtained by using the verification set is taken as a final model; A user frames a target of interest in the image I, and a corresponding feature block in a feature matrix F of the image is found as a template; The image I has a size of n*n, and the feature matrix F has a size of c*m*m, wherein c is a channel number of the feature matrix, and m is a width and a height of the feature matrix; Similarities of each part are calculated one by one in the feature matrix F of the whole image by using a sliding window method, and if the similarity is greater than a set threshold value, the feature block is considered to be a molecule of the same type as the target molecule, otherwise, the feature block is moved to a next position for calculation until the whole feature layer is traversed, wherein the threshold value θ can be adjusted according to a visual requirement of the user. The characteristic matrix FB of the target molecule is slid and matched in the characteristic matrix F of the whole molecule image in the order from left to right and from top to bottom, and the step length of horizontal direction and vertical direction is s h and s v , wherein s h ∈(0,4], s v ∈(0,4], s h and s v are all integers; Let the feature block corresponding to the feature matrix FB of the target molecule in the feature matrix F be FB k , where k represents the number of movements, and let the upper left corner coordinates of FB k be (p k , q k ) Coordinates of the same type of target are converted from a feature space to coordinates on an image, and a non-maximum suppression method is used to remove overlapping frames to obtain the same type of target. and where ||FB|| and ||FB k || are the norms of FB and FB k , respectively, and then the cosine distance is used to calculate the similarity of FB' and FB k as one-dimensional vectors, as follows: The same type of target is represented as follows: A non-maximum suppression method is used to select a candidate region most similar to the feature matrix FB of the target molecule from multiple candidate regions as a molecule of the same type as the target molecule, so that multiple accurate targets are obtained. The feature block corresponding to the same kind of molecule as the target molecule is mapped to the molecular image to form a candidate region, with the upper left corner coordinates being (i k ,j k ), the width and height being (w F ,h F ); In a dynamic image, the same type of target is tracked dynamically. The user frames a target of interest in the original image, which comprises the following steps: A user uses a mouse to frame a target of interest to obtain IB, and records a top-left corner coordinate (i, j) and a width and a height (w0, h0) of the frame.

2. The human-machine interaction based target molecule observation method according to claim 1, characterized in that: The corresponding feature block in the feature matrix of the image is found as a template, which comprises the following steps: In the dynamic image, the multiple targets are tracked, which comprises the following steps:

3. The human-machine interaction based target molecule observation method according to claim 2, characterized in that: After the image of the first frame is processed by using the method in claims 1 to 3, the same type of target is detected, and a CSR-DCF tracking algorithm is used to track the target in each frame of the image. The selected target molecule is mapped into the feature matrix F to obtain the feature matrix FB of the target molecule. According to the proportional relationship between the picture I and the feature map F, the coordinates of the upper left corner of FB in the feature map are (p, q), the width and height are (w, h), and are represented as follows: FB = F[p: p+w, q: q+h] (1) F F )​ 4. The target molecule observation method based on human-computer interaction according to any one of claims 1 to 3, characterized by: ​ ​

Images