Primary lysosome and secondary lysosome classification method and system based on image recognition

Abstract

The present application relates to the field of image processing technology and discloses a method and system for classifying primary and secondary lysosomes based on image recognition. The method comprises: obtaining time series images containing lysosomes, outputting a predicted probability map of the presence of lysosomes in the time series images based on a recognition model; determining candidate locations of the lysosomes based on the predicted probability map, determining whether there is overlap at the candidate locations, and if so, defining the candidate locations as overlapping regions; obtaining a light field image of the overlapping region, segmenting the overlapping region into multiple independent regions based on the light field image, and obtaining the three-dimensional coordinates of the lysosomes in each independent region; identifying the same lysosome in multiple frames of continuous time series images based on the three-dimensional coordinates, generating corresponding motion trajectories, establishing a classification model based on the motion trajectory dataset of the primary and secondary lysosomes, and identifying the type of lysosomes based on the classification model. The present application improves the efficiency of lysosome classification.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for classifying primary lysosomes and secondary lysosomes based on image recognition. Background Art

[0002] Lysosomes are key organelles within cells, responsible for degrading and recycling biological macromolecules, aging organelles, and foreign pathogens within the cell. Lysosomes are mainly divided into primary lysosomes and secondary lysosomes. Traditional lysosome identification and classification methods mainly include biochemical labeling, electron microscopy observation, and live cell imaging using fluorescently labeled lysosome-specific proteins. In recent years, with the rapid development of computer vision and deep learning technologies, cell classification methods based on image recognition have gradually become a research hotspot.

[0003] For example, the Chinese patent application with publication number CN119314173A discloses a cell classification and analysis device and method based on super-resolution image mitochondrial subtypes. The method includes: step 1, based on the segmentation image of all mitochondria in the cell, calculating the characteristics of each mitochondria and performing dimensionality reduction processing, clustering the mitochondrial characteristics after dimensionality reduction to obtain the category of each mitochondria; step 2, based on the cell and the category of each mitochondria obtained in step 1, calculating the proportion of each type of mitochondria in the cell in which it is located, determining the characteristics of the cell and performing dimensionality reduction processing, clustering the cell characteristics after dimensionality reduction to obtain the category of the cell.

[0004] However, the above methods are mainly based on static image analysis, ignoring the dynamic changes of lysosomes in time series. In addition, lysosomes may overlap or aggregate within cells, making detection and classification difficult. Mitochondrial classification methods based on super-resolution images mainly analyze single mitochondria and lack the ability to process overlapping or aggregated objects. Summary of the Invention

[0005] To solve the above technical problems, the present application provides a primary lysosome and secondary lysosome classification method and system based on image recognition, which is used to improve the accuracy of primary lysosome and secondary lysosome classification.

[0006] In a first aspect, the present application provides a method for classifying primary lysosomes and secondary lysosomes based on image recognition, the method comprising:

[0007] Step S1: acquiring multiple frames of continuous time series images containing lysosomes, establishing a recognition model, and outputting a predicted probability map of the presence of lysosomes in the time series images based on the recognition model;

[0008] Step S2: identifying the peak position in the predicted probability map, defining the peak position as a candidate position of the lysosome in the time series image, and determining whether there is an overlap at the candidate position. If so, defining the candidate position as an overlapping area;

[0009] Step S3: acquiring a light field image at the overlapping region, dividing the overlapping region into a plurality of independent regions based on the light field image, and acquiring the three-dimensional coordinates of the lysosome in each independent region;

[0010] Step S4: Based on the three-dimensional coordinates, the same lysosome in multiple frames of the time series images is identified, the dynamic motion trajectory of the same lysosome is generated, a classification model is established, and a motion trajectory dataset with primary lysosomes and secondary lysosomes respectively labeled is obtained. The motion trajectory dataset is used as training data for the classification model, and the type of the lysosome is identified based on the classification model.

[0011] In combination with the first aspect, in a first implementation of the first aspect of the present application, outputting a predicted probability map of the presence of lysosomes in the time series image includes:

[0012] Obtain a first cell image labeled with primary lysosomes or secondary lysosomes, establish a recognition model based on a neural network model, train the recognition model based on the first cell image, input the time series image into the trained recognition model, output a prediction probability map, the prediction probability map includes the prediction probability of the existence of lysosomes at each pixel in the time series image, identify the position information and size information of each lysosome in the first cell image, and label it to obtain a second cell image, calculate the true score map of the existence of the lysosome based on the second cell image, calculate the deviation value between the prediction probability map and the true score map, if the deviation value is greater than a first threshold, adjust the parameters of the recognition model, the recognition model obtains a new prediction probability map based on the adjusted parameters, and repeat this operation until the deviation value is less than or equal to the first threshold.

[0013] In combination with the first aspect, in a second implementation of the first aspect of the present application, calculating the true score map of the presence of the lysosome based on the second cell image includes:

[0014] The basic parameters of the Gaussian distribution function are set based on the position information and the size information, a Gaussian distribution function is generated for each lysosome based on the basic parameters, the Gaussian value of each pixel of each lysosome in the time series image is calculated based on the Gaussian distribution function, and the Gaussian values ​​of all lysosomes at the same pixel position in the time series image are superimposed to obtain a true score map.

[0015] In combination with the first aspect, in a third implementation of the first aspect of the present application, determining whether there is an overlap at the candidate positions includes:

[0016] Calculate the distance value between the candidate positions in the time series image; if the distance value is less than a second threshold, define the corresponding two candidate positions as target positions; binarize the predicted probability map corresponding to the target position; identify the lysosome region from the time series image based on the binarization result; calculate the regional contour value of the lysosome region; if the regional contour value is less than a third threshold, it indicates that there is overlap among the target positions.

[0017] In combination with the first aspect, in a fourth implementation of the first aspect of the present application, segmenting the overlapping area into multiple independent areas based on the light field image includes:

[0018] The light field image is decomposed into multiple viewpoint images, each viewpoint image includes lysosome information under different viewpoints, the disparity information between different viewpoint images is calculated, the position information of each lysosome in the overlapping area in the depth direction is estimated based on the disparity information, the overlapping area is divided into different depth layers based on the position information, each depth layer is refocused to generate an independent area containing only one lysosomal cell, the center of gravity coordinates of the independent area are calculated based on the depth information of the lysosome in the independent area, and the three-dimensional coordinates of the lysosome are obtained based on the center of gravity coordinates.

[0019] In combination with the first aspect, in a fifth implementation of the first aspect of the present application, identifying the same lysosome in multiple frames of continuous time series images based on the three-dimensional coordinates includes:

[0020] Lysosomes in the time series images of the first frame and the second frame are defined as the first lysosome and the second lysosome, respectively. An identifier is defined for each first lysosome. Trajectory information is initialized based on the three-dimensional coordinates of the first lysosome. The distance value between the second lysosome in the time series image of the second frame and all the first lysosomes in the first frame is calculated. The first lysosome whose distance value is less than a fourth threshold is defined as the target lysosome. The cosine value of the angle between the movement directions of the target lysosome and the second lysosome is calculated. If the cosine value of the angle is less than a fifth threshold, the second lysosome and the target lysosome are the same lysosome; otherwise, the second lysosome is a new lysosome. An identifier is set for the new lysosome. Time series images of the new lysosome are obtained. The dynamic trajectory of the new lysosome is tracked based on the time series images.

[0021] In combination with the first aspect, in a sixth implementation of the first aspect of the present application, generating a dynamic motion trajectory of the same lysosome includes:

[0022] The three-dimensional coordinates of the second lysosome that is the same as the first lysosome in the second frame of the time series image are added to the trajectory information of the first lysosome. In the third frame of the time series image, if the same lysosome is not detected, but the same lysosome is detected in the fourth frame of the time series, the same lysosome in the third frame of the time series image is interpolated and completed.

[0023] In combination with the first aspect, in a seventh implementation of the first aspect of the present application, identifying the type of the lysosome based on the classification model includes:

[0024] The dynamic features of the lysosome are extracted from the trajectory information, and the dynamic features include the displacement velocity, the rate of change of the movement direction, the standard deviation of the lysosome in the depth direction, and the average time that the lysosome stays at each position of the same lysosome in adjacent time series images. A comprehensive feature vector is formed based on the dynamic features of each lysosome. A classification model is established based on the machine learning model. Feature vectors are respectively extracted from the motion trajectory data set labeled as primary lysosomes and secondary lysosomes. The feature vectors are input into the classification model as training data, and the comprehensive feature vector is input into the trained classification model. The classification model outputs the type of the lysosome.

[0025] In a second aspect, the present application provides a primary lysosome and secondary lysosome classification system based on image recognition, the system comprising:

[0026] a recognition module, configured to acquire multiple frames of continuous time series images containing lysosomes, establish a recognition model, and output a predicted probability map of the presence of lysosomes in the time series images based on the recognition model;

[0027] an overlap determination module, configured to identify a peak position in the predicted probability map, define the peak position as a candidate position of a lysosome in the time series image, determine whether there is an overlap at the candidate position, and if so, define the candidate position as an overlap region;

[0028] a segmentation module, configured to obtain a light field image at the overlapping region, segment the overlapping region into a plurality of independent regions based on the light field image, and obtain the three-dimensional coordinates of a lysosome in each of the independent regions;

[0029] A classification module is used to identify the same lysosome in multiple frames of continuous time series images based on the three-dimensional coordinates, generate a dynamic motion trajectory of the same lysosome, establish a classification model, obtain a motion trajectory dataset with primary lysosomes and secondary lysosomes labeled respectively, use the motion trajectory dataset as training data for the classification model, and identify the type of the lysosome based on the classification model.

[0030] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0031] The present invention establishes a recognition model based on a neural network, outputs the probability value of the existence of a lysosome at each pixel based on the recognition model, generates a Gaussian function formula for each lysosome, obtains the corresponding true score map, and judges the prediction accuracy of the recognition model by calculating the deviation between the true score map and the predicted probability map, thereby improving the recognition ability of the recognition model; after recognition, it is judged whether there is overlap between the candidate positions. The present invention first calculates the distance between the two candidate positions, first screens out the candidate position pairs whose distance is less than a preset threshold, and then calculates the contour value at the candidate position. The contour value is used to evaluate the shape characteristics of the lysosome. If the contour value does not meet the preset label, it indicates that there is overlap at the candidate position.

[0032] The present invention also uses light field images to divide the overlapping area into multiple independent areas. Mainly starting from the depth direction of the lysosome, the lysosome is divided into different depth layers, and each depth layer is refocused to generate an independent area containing only one lysosomal cell. By calculating the center of gravity coordinates, the three-dimensional coordinates of the lysosome can be accurately located; by comparing the distance and movement direction of the lysosome, it is confirmed whether the lysosomes in consecutive image frames are the same lysosome, and the position information of the lysosome is added to the trajectory information, which can achieve accurate identification and tracking of the lysosome; by using the motion trajectory to train the classification model, the lysosome is divided into primary lysosomes and secondary lysosomes, thereby improving the classification efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 Schematic diagram of an embodiment of a method for classifying primary lysosomes and secondary lysosomes based on image recognition in an embodiment of the present application;

[0035] Figure 2 Schematic diagram of the prediction probability map in the embodiment of the present application;

[0036] Figure 3 This is a flow chart for identifying the same lysosome in the examples of this application;

[0037] Figure 4 This is a schematic diagram of an embodiment of the primary lysosome and secondary lysosome classification system based on image recognition in the embodiments of the present application. DETAILED DESCRIPTION

[0038] The present application embodiment provides a kind of primary lysosome and secondary lysosome classification method and system based on image recognition.The term " first ", " second ", " third ", " fourth " etc. (if any) in the specification and claims of the present application and the above-mentioned accompanying drawing are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. Should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiment described here can be implemented in the order except the content illustrated or described here. In addition, the term " comprises " or " has " and any distortion thereof, are intended to cover non-exclusive inclusion, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0039] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 In the embodiments of the present application, an embodiment of the method for classifying primary lysosomes and secondary lysosomes based on image recognition includes:

[0040] Step S1: Acquire multiple frames of continuous time series images containing lysosomes, establish a recognition model, and output a predicted probability map of the presence of lysosomes in the time series images based on the recognition model.

[0041] Specifically, imaging equipment such as a confocal microscope is used to acquire cell image data containing lysosomes. Multiple frames of images are captured continuously to form a time series image. The acquired image data is then preprocessed to improve data quality and reduce the impact of noise. A recognition model is constructed based on a deep learning model, such as U-Net. This recognition model uses a pretrained convolutional neural network as a feature extractor and a sigmoid activation function to output the probability of the presence of a lysosome at each pixel in each frame of the time series image, forming a predicted probability map.

[0042] Step S2: Identify the peak position in the prediction probability map, define the peak position as the candidate position of the lysosome in the time series image, and determine whether there is an overlap at the candidate position. If so, define the candidate position as the overlapping area.

[0043] Specifically, if Figure 2As shown, it is a schematic diagram of the prediction probability map, comparing each pixel with the pixel value in the preset neighborhood. If the pixel value is greater than all the neighboring pixel values, it is marked as the peak position, such as the position of A. There may be one or more peak positions in a prediction probability map. The centroid of the filtered peak area is calculated, and the centroid position is used as the candidate position of the lysosome. If the probability value at the peak position is higher than the preset confidence threshold (such as 0.8 or 0.9), it can be preliminarily considered that the possibility of the presence of lysosomes at this position is high. However, in the stage of active cell metabolism, such as during cell phagocytosis, autophagy or stress response, a large number of lysosomes are generated in the cell, resulting in an increase in the lysosome density. The high-density lysosomes are distributed in a limited space, which easily leads to overlap between lysosomes. The overlap will reduce the accuracy of identifying the lysosomal area and type, because it is necessary to judge whether the lysosomes at the candidate position overlap or whether the lysosomes overlap with other cells. The specific judgment method will be explained later.

[0044] Step S3: Acquire a light field image at the overlapping region, segment the overlapping region into multiple independent regions based on the light field image, and acquire the three-dimensional coordinates of the lysosome in each independent region.

[0045] Specifically, a light-field microscope (LFM) was used to re-image the cells in the overlapping area to obtain light-field images. Each light-field image contains elemental images from multiple viewpoints, and each elemental image contains lysosomal information from different angles. Using the multi-viewpoint information in the light-field images, the lysosomes in the overlapping area were reconstructed in three dimensions, and the corresponding three-dimensional coordinates were obtained. The overlapping lysosomes were then separated into multiple independent regions in three-dimensional space, each containing a lysosome. Separation using depth information (in the Z-axis direction) can effectively solve the recognition and classification problems caused by overlap, ensuring its accuracy.

[0046] Step S4: Identify the same lysosome in multiple frames of continuous time series images based on the three-dimensional coordinates, generate the dynamic motion trajectory of the same lysosome, establish a classification model, obtain a motion trajectory dataset with primary lysosomes and secondary lysosomes labeled respectively, use the motion trajectory dataset as training data for the classification model, and identify the type of lysosome based on the classification model.

[0047] Specifically, a unique trajectory ID is assigned to each lysosome detected in the initial frame (t=1). For example, if five lysosomes are detected at time t=1, IDs T1 to T5 are assigned. For images at subsequent time points (t=2, 3, ...), a nearest neighbor search or Kalman filter is used to match the lysosome at the current time point with the lysosome at the previous time point to determine whether they are the same lysosome. If so, the matched lysosome position is added to the corresponding trajectory information, updating the lysosome's motion trajectory. By identifying the same lysosome, the positions of lysosomes at different time points are correctly associated, avoiding trajectory breakage or incorrect association.

[0048] Based on known biological knowledge, the motion trajectory dataset is labeled based on the multi-dimensional characteristics of primary and secondary lysosomes, such as movement patterns, speed, and direction. Appropriate machine learning algorithms (such as random forests, support vector machines, and neural networks) are selected to build a classification model. By using the motion trajectory to train the classification model, lysosomes are divided into primary and secondary lysosomes, thereby improving the classification efficiency and accuracy.

[0049] In a specific embodiment, outputting a predicted probability map of the presence of lysosomes in a time series image specifically includes the following steps:

[0050] A first cell image labeled with primary lysosomes or secondary lysosomes is obtained, a recognition model is established based on a neural network model, the recognition model is trained based on the first cell image, and the time series image is input into the trained recognition model to output a prediction probability map, the prediction probability map including the prediction probability of the existence of lysosomes at each pixel in the time series image, the position information and size information of each lysosome in the first cell image is identified and labeled to obtain a second cell image, a true score map of the existence of lysosomes is calculated based on the second cell image, and a deviation value between the prediction probability map and the true score map is calculated. If the deviation value is greater than a first threshold value, the parameters of the recognition model are adjusted, and the recognition model obtains a new prediction probability map based on the adjusted parameters, and this operation is repeated until the deviation value is less than or equal to the first threshold value.

[0051] Specifically, an image annotation tool is used to annotate the lysosome type in the image to obtain a first cell image, a recognition model is constructed based on a neural network model such as U-Net, and the recognition model is trained using the first cell image. The recognition model extracts the position features of the primary lysosomes and secondary lysosomes in the first cell image, and the time series image is input into the trained recognition model. The recognition model outputs a prediction probability map for each time series image. The prediction probability map is a two-dimensional array. Each position in the array represents the probability of the existence of a lysosome at the pixel position. The probability value is between [0, 1]. The closer the probability value is to 1, the more likely the model believes that a primary lysosome or a secondary lysosome exists at the position.

[0052] In order to further improve the prediction accuracy of the neural network, it is necessary to correct the predicted probability. The present invention obtains the position information (the location coordinates and other tissue cell information within a preset range) and size information (size, shape, etc.) of each lysosome in the first cell image, and generates a true score map based on the second cell image. The true score map is also a two-dimensional matrix, in which the value of each pixel represents the true score of the target object at the pixel position. The true score map of the present invention is obtained using a Gaussian function. The specific generation process will be explained later. By comparing the deviation between the predicted probability map and the true score map, the prediction accuracy of the recognition model can be evaluated. If the deviation between the predicted probability map and the true score map is large, it is necessary to adjust the model parameters, such as the learning rate, weight decay coefficient, etc. By rationally utilizing the true score map, the recognition model of the lysosome can be effectively trained and optimized to improve the prediction accuracy and reliability.

[0053] In a specific embodiment, calculating a true score map of lysosome presence based on the second cell image specifically includes the following steps:

[0054] The basic parameters of the Gaussian distribution function are set based on the position information and size information. A Gaussian distribution function is generated for each lysosome based on the basic parameters. The Gaussian value of each pixel of each lysosome in the time series image is calculated based on the Gaussian distribution function. The Gaussian values ​​of all lysosomes at the same pixel position in the time series image are superimposed to obtain the true score map.

[0055] Specifically, the pixel coordinates (x0, y0) and size information (e.g., diameter D) of each lysosome are extracted from the annotated data of the second cell image, and the standard deviation σ of the Gaussian distribution is calculated using Formula 1: For each lysosome, the corresponding Gaussian distribution function is generated according to its center coordinates and standard deviation. The Gaussian distribution function is: A is the amplitude of the Gaussian distribution. By calculating the Gaussian distribution function for each lysosome, its spatial distribution in the image can be accurately represented. For each pixel position (x0, y0) in the image, the corresponding Gaussian value G(x, y) is calculated and stored in a two-dimensional matrix with the same size as the image. For each pixel position in the image, the Gaussian values ​​of all lysosomes at that position are superimposed. For example, in the second cell image, there are three lysosomes: lysosome A, lysosome B, and lysosome C. The Gaussian values ​​at pixel coordinates (10, 10) are 1, 0.1, and 0.002, respectively. The superposition result is S(10, 10) = 1 + 0.1 + 0.002 = 1.102. By superimposing the Gaussian values ​​of all lysosomes, a comprehensive probability map can be obtained, reflecting the probability of the existence of a lysosome at each pixel position. When multiple lysosomes are close to or overlap in space, their probability distributions will superimpose on each other, forming a higher probability value.

[0056] In a specific embodiment, determining whether there is overlap at the candidate positions specifically includes the following steps:

[0057] The distance between candidate positions in the time series image is calculated. If the distance is less than the second threshold, the corresponding two candidate positions are defined as target positions. The predicted probability map corresponding to the target position is binarized. Based on the binarization result, the lysosome region is identified from the time series image. The regional contour value of the lysosome region is calculated. If the regional contour value is less than the third threshold, it indicates that there is overlap in the target positions.

[0058] Specifically, when lysosomes overlap in the image, their morphological characteristics and intensity distribution may change, making it difficult for the classification model to accurately identify their types (primary / secondary lysosomes). When making overlap judgments, first determine whether any two candidate positions in the time series image are close to each other. If so, it indicates that the probability of overlap is high. If they are not close to each other, there is no need to proceed to the next step. For the judgment of the close state, the distance value between the two candidate positions in the time series image can be calculated. Suppose the distance value D between candidate position 1 and candidate position 2 is calculated. 12 , all the distance values ​​D calculated 12 Compare with the preset second threshold (e.g. 5 pixels), if D 12 If the distance is less than the second threshold, the candidate positions (candidate position 1 and candidate position 2) are marked as the target position pair (1, 2). By distance calculation, the candidate position pairs that may overlap can be quickly screened out, reducing the computational complexity of subsequent processing.

[0059] The predicted probability map corresponding to the target position (candidate position 1 and candidate position 2) is converted into a binary map, and the probability value is converted into a binary value (0 or 1). The connected region analysis is performed on the binary image to identify all connected regions. The possible lysosome regions are screened out based on the area, shape and other features. The boundary contour of each lysosome region is extracted, and the contour value C of each lysosome region is calculated based on Formula 2. Formula 2 is Here, C represents the circularity of the lysosomal region. The calculated contour value is compared with a preset third threshold, which is set based on the average shape characteristics of the lysosomes, for example, the circularity threshold is set to 0.5. Overlapping lysosomes typically have irregular shapes, resulting in lower contour values. Therefore, if the contour value is less than the third threshold, the lysosomal region is considered to be overlapping. The above steps are repeated for all candidate locations to identify all overlapping lysosomal regions. By identifying overlapping regions, the overlapping lysosomes can be separated and their features extracted, thereby improving classification accuracy.

[0060] In a specific embodiment, segmenting the overlapping area into multiple independent areas based on the light field image specifically includes the following steps:

[0061] The light field image is decomposed into multiple viewpoint images, each viewpoint image includes lysosome information under different viewpoints, the disparity information between different viewpoint images is calculated, the position information of each lysosome in the overlapping area in the depth direction is estimated based on the disparity information, the overlapping area is divided into different depth layers based on the position information, each depth layer is refocused to generate an independent area containing only one lysosomal cell, the center of gravity coordinates of the independent area are calculated based on the depth information of the lysosome in the independent area, and the three-dimensional coordinates of the lysosome are obtained based on the center of gravity coordinates.

[0062] Specifically, a light field microscope (LFM) is used to obtain a light field image of the overlapping area. The light field image contains sub-images from different microlenses, each sub-image corresponds to a different viewpoint, and the light field image is decomposed into multiple viewpoint images, each viewpoint image corresponds to a microlens. Feature points are matched between different viewpoint images, for example, using SIFT, SURF or ORB algorithms, and disparity is calculated based on the matched feature points, that is, the position difference of corresponding pixels in different viewpoint images. The calculated disparity value is stored in the disparity map to form a two-dimensional matrix. The depth information of each lysosome is calculated using the triangulation method or the depth estimation formula of light field imaging through the disparity information. The depth estimation formula is: Where z is the depth, f is the focal length, b is the baseline length, and d is the disparity. Through depth estimation, the position information of the lysosome in the depth direction can be obtained. The lysosome is divided into different depth layers using a depth threshold or clustering algorithm. If the depth range is [0, 10], the depth layers can be divided into [0, 3], [3, 6], and [6, 10].

[0063] Each depth layer is divided into regions, overlapping lysosomes are assigned to different depth layers, and each depth layer is refocused using a refocusing algorithm of light field imaging. The element image (EI) is moved and superimposed to the corresponding depth position, and each refocused image is segmented to extract the lysosome region and generate an independent lysosome region. For example, an image segmentation algorithm (such as threshold segmentation, region growing, edge detection) is used to extract the lysosome region. For the lysosomes in each independent region, the center of gravity coordinates are calculated. By calculating the center of gravity coordinates, the position of the lysosome can be accurately determined. The present invention can accurately separate overlapping lysosomes and improve the accuracy of lysosome classification and tracking through light field imaging technology and depth estimation.

[0064] In a specific embodiment, identifying the same lysosome in multiple frames of continuous time series images based on three-dimensional coordinates specifically includes the following steps:

[0065] The lysosomes in the time series images of the first frame and the second frame are defined as the first lysosome and the second lysosome, respectively. An identifier is defined for each first lysosome. The trajectory information is initialized based on the three-dimensional coordinates of the first lysosome. The distance between the second lysosome in the time series image of the second frame and all the first lysosomes in the first frame is calculated. The first lysosome whose distance value is less than a fourth threshold is defined as the target lysosome. The cosine value of the angle between the movement directions of the target lysosome and the second lysosome is calculated. If the cosine value of the angle is less than a fifth threshold, the second lysosome and the target lysosome are the same lysosome; otherwise, it is a new lysosome. An identifier is set for the new lysosome, time series images of the new lysosome are obtained, and the dynamic trajectory of the new lysosome is tracked based on the time series images.

[0066] Specifically, if Figure 3 As shown in the figure, it is a flow chart for identifying the same lysosome. Assume that there are 5 time series images, and there are 3 lysosomes in the first frame, marked as L. 11 , L 12 , L 13 , there are 4 lysosomes in the second frame, marked as L 21 , L 22 , L 23 and L 24Initialize a trajectory information for each first lysosome, record its initial three-dimensional coordinates and timestamp, calculate the Euclidean distance between each second lysosome and all first lysosomes in the first frame, and for each second lysosome, select the first lysosome with a distance value less than the fourth threshold (e.g., 5 pixels) as the target lysosome. If a second lysosome does not have a first lysosome with a distance less than the fourth threshold, mark it as a new lysosome, calculate the movement direction vector of the target lysosome and the second lysosome, and calculate the cosine value of the angle between the movement direction vectors of the target lysosome and the second lysosome. The closer the cosine value of the angle is to 1, the more consistent the movement direction is. Therefore, if the cosine value of the angle is greater than the fifth threshold (e.g., 0.9), it is considered that the second lysosome and the target lysosome are the same lysosome. By comparing the movement directions, the matching relationship of the lysosomes can be further confirmed, thereby improving the accuracy of the matching.

[0067] Initialize a new trajectory for the new lysosome, record its initial three-dimensional coordinates and timestamp, track the new lysosome, repeat steps 1 to 3, and record its motion trajectory. The present invention can accurately identify and track lysosomes through the lysosome identification and tracking method based on three-dimensional coordinates.

[0068] In a specific embodiment, generating the dynamic motion trajectory of the same lysosome specifically includes the following steps:

[0069] The three-dimensional coordinates of the second lysosome that is the same as the first lysosome in the second frame of the time series image are added to the trajectory information of the first lysosome. In the third frame of the time series image, if the same lysosome is not detected, but the same lysosome is detected in the fourth frame of the time series, the same lysosome in the third frame of the time series image is interpolated and completed.

[0070] Specifically, based on distance calculation and motion direction analysis, it is determined whether the two lysosomes are the same lysosome, and the three-dimensional coordinates and timestamps of the matched lysosomes are added to their corresponding trajectory information. When a trajectory interruption is detected, if no lysosome matching L_1 is detected at time t=3, the cause of the interruption is analyzed, and the interrupted time points are interpolated and supplemented, such as using the positions of the previous and next time points for linear interpolation, and the interpolated positions are added to the trajectory to fill the gaps in the interruption. The trajectory is smoothed using methods such as Kalman filtering, spline interpolation, or moving average. The present invention combines motion prediction, interpolation and smoothing technology to effectively address various problems in the lysosome tracking process and ensure the integrity and accuracy of the trajectory.

[0071] In a specific embodiment, identifying the type of lysosome based on the classification model specifically includes the following steps:

[0072] The dynamic features of lysosomes are extracted from the trajectory information. The dynamic features include the displacement velocity, the rate of change of the motion direction, the standard deviation of the lysosome in the depth direction, and the average time the lysosome stays at each position of the same lysosome in adjacent time series images. A comprehensive feature vector is formed based on the dynamic features of each lysosome. A classification model is established based on the machine learning model. Feature vectors are extracted from the motion trajectory datasets labeled as primary lysosomes and secondary lysosomes respectively. The feature vectors are input into the classification model as training data, and the comprehensive feature vector is input into the trained classification model. The classification model outputs the type of lysosome.

[0073] Specifically, the displacement of the same lysosome between adjacent time points is calculated based on the trajectory information, and its speed is calculated (primary lysosomes usually move slowly in the cytoplasm at a lower speed, and secondary lysosomes usually show a higher speed and a more active movement pattern), as well as the rate of change of movement direction (primary lysosomes mainly move slowly in the cytoplasm, and their movement direction is less affected by the intracellular environment. Therefore, the movement direction of primary lysosomes is relatively stable and the direction changes little, while secondary lysosomes need to contact different organelles or substances, so the movement direction of secondary lysosomes may change frequently). , showing random motion or Brownian motion characteristics), the standard deviation of the position change of lysosomes in the depth direction (Z axis) (primary lysosomes are usually distributed in the cytoplasm, and the depth variation is small; secondary lysosomes need to contact organelles or substances at different depths, such as fusing with endosomes or participating in autophagy, resulting in large depth variations), and residence time (primary lysosomes need to obtain hydrolases and other substances from the Golgi apparatus and therefore need to stay around it for a longer time; secondary lysosomes participate in the endoplasmic reticulum autophagy (ER-phagy) process and need to fuse with the endoplasmic reticulum and degrade its contents).

[0074] The extracted dynamic features are fused into a comprehensive feature vector, and a machine learning classification model is constructed using a machine learning algorithm (such as a deep neural network). The feature vectors labeled as primary lysosomes and secondary lysosomes are used as training data. The machine learning model extracts the features of primary lysosomes and secondary lysosomes for learning, and the feature vectors are input into the trained classification model to obtain the classification results.

[0075] The above describes the primary lysosome and secondary lysosome classification method based on image recognition in the embodiment of the present application. The following describes the primary lysosome and secondary lysosome classification system based on image recognition in the embodiment of the present application. Figure 4 In one embodiment of the present application, a primary lysosome and secondary lysosome classification system based on image recognition includes:

[0076] A recognition module is used to obtain multiple frames of continuous time series images containing lysosomes, establish a recognition model, and output a predicted probability map of the presence of lysosomes in the time series images based on the recognition model;

[0077] The overlap judgment module is used to identify the peak position in the predicted probability map, define the peak position as the candidate position of the lysosome in the time series image, and judge whether there is overlap at the candidate position. If so, the candidate position is defined as the overlapping area;

[0078] a segmentation module, configured to obtain a light field image at the overlapping region, segment the overlapping region into a plurality of independent regions based on the light field image, and obtain the three-dimensional coordinates of the lysosomes in each independent region;

[0079] The classification module is used to identify the same lysosome in multiple frames of continuous time series images based on three-dimensional coordinates, generate the dynamic motion trajectory of the same lysosome, establish a classification model, obtain a motion trajectory dataset with primary lysosomes and secondary lysosomes labeled separately, use the motion trajectory dataset as training data for the classification model, and identify the type of lysosome based on the classification model.

[0080] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0082] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for classifying primary lysosomes and secondary lysosomes based on image recognition, characterized in that: The method comprises: Step S1: acquiring multiple frames of continuous time series images containing lysosomes, establishing a recognition model, and outputting a predicted probability map of the presence of lysosomes in the time series images based on the recognition model; Step S2: identifying the peak position in the predicted probability map, defining the peak position as a candidate position of the lysosome in the time series image, and determining whether there is an overlap at the candidate position. If so, defining the candidate position as an overlapping area; Step S3: acquiring a light field image at the overlapping region, dividing the overlapping region into a plurality of independent regions based on the light field image, and acquiring the three-dimensional coordinates of the lysosome in each independent region; Step S4: identifying the same lysosome in multiple frames of the time series images based on the three-dimensional coordinates, generating a dynamic motion trajectory of the same lysosome, establishing a classification model, obtaining a motion trajectory dataset with primary lysosomes and secondary lysosomes respectively labeled, using the motion trajectory dataset as training data for the classification model, and identifying the type of the lysosome based on the classification model; Outputting the predicted probability map of the presence of lysosomes in the time series image in step S1 includes the following steps: Obtain a first cell image labeled with primary lysosomes or secondary lysosomes, establish a recognition model based on a neural network model, train the recognition model based on the first cell image, input the time series image into the trained recognition model, output a prediction probability map, the prediction probability map includes the prediction probability of the existence of lysosomes at each pixel in the time series image, identify the position information and size information of each lysosome in the first cell image, and label it to obtain a second cell image, calculate the true score map of the existence of the lysosome based on the second cell image, calculate the deviation value between the prediction probability map and the true score map, if the deviation value is greater than a first threshold, adjust the parameters of the recognition model, the recognition model obtains a new prediction probability map based on the adjusted parameters, and repeat this operation until the deviation value is less than or equal to the first threshold.

2. The method according to claim 1, characterized in that Calculating the true score map of lysosome presence based on the second cell image comprises the following steps: The basic parameters of the Gaussian distribution function are set based on the position information and the size information, a Gaussian distribution function is generated for each lysosome based on the basic parameters, the Gaussian value of each pixel of each lysosome in the time series image is calculated based on the Gaussian distribution function, and the Gaussian values ​​of all lysosomes at the same pixel position in the time series image are superimposed to obtain a true score map.

3. The method according to claim 1, characterized in that Determining whether there is overlap at the candidate positions includes the following steps: Calculate the distance value between the candidate positions in the time series image; if the distance value is less than a second threshold, define the corresponding two candidate positions as target positions; binarize the predicted probability map corresponding to the target position; identify the lysosome region from the time series image based on the binarization result; calculate the regional contour value of the lysosome region; if the regional contour value is less than a third threshold, it indicates that there is overlap among the target positions.

4. The method according to claim 3, characterized in that Segmenting the overlapping area into a plurality of independent areas based on the light field image comprises the following steps: The light field image is decomposed into multiple viewpoint images, each viewpoint image includes lysosome information under different viewpoints, the disparity information between different viewpoint images is calculated, the position information of each lysosome in the overlapping area in the depth direction is estimated based on the disparity information, the overlapping area is divided into different depth layers based on the position information, each depth layer is refocused to generate an independent area containing only one lysosomal cell, the center of gravity coordinates of the independent area are calculated based on the depth information of the lysosome in the independent area, and the three-dimensional coordinates of the lysosome are obtained based on the center of gravity coordinates.

5. The method according to claim 1, wherein Identifying the same lysosome in multiple frames of the time series images based on the three-dimensional coordinates comprises the following steps: Lysosomes in the time series images of the first frame and the second frame are defined as the first lysosome and the second lysosome, respectively. An identifier is defined for each first lysosome. Trajectory information is initialized based on the three-dimensional coordinates of the first lysosome. The distance value between the second lysosome in the time series image of the second frame and all the first lysosomes in the first frame is calculated. The first lysosome whose distance value is less than a fourth threshold is defined as the target lysosome. The cosine value of the angle between the movement directions of the target lysosome and the second lysosome is calculated. If the cosine value of the angle is less than a fifth threshold, the second lysosome and the target lysosome are the same lysosome; otherwise, the second lysosome is a new lysosome. An identifier is set for the new lysosome. Time series images of the new lysosome are obtained. The dynamic trajectory of the new lysosome is tracked based on the time series images.

6. The method according to claim 5, characterized in that Generating the dynamic motion trajectory of the same lysosome includes the following steps: The three-dimensional coordinates of the second lysosome that is the same as the first lysosome in the second frame of the time series image are added to the trajectory information of the first lysosome. In the third frame of the time series image, if the same lysosome is not detected, but the same lysosome is detected in the fourth frame of the time series, the same lysosome in the third frame of the time series image is interpolated and completed.

7. The method according to claim 6, characterized in that Identifying the type of the lysosome based on the classification model comprises the following steps: The dynamic features of the lysosome are extracted from the trajectory information, and the dynamic features include the displacement velocity, the rate of change of the movement direction, the standard deviation of the lysosome in the depth direction, and the average time that the lysosome stays at each position of the same lysosome in adjacent time series images. A comprehensive feature vector is formed based on the dynamic features of each lysosome. A classification model is established based on the machine learning model. Feature vectors are respectively extracted from the motion trajectory data set labeled as primary lysosomes and secondary lysosomes. The feature vectors are input into the classification model as training data, and the comprehensive feature vector is input into the trained classification model. The classification model outputs the type of the lysosome.

8. A primary lysosome and secondary lysosome classification system based on image recognition, used to implement the primary lysosome and secondary lysosome classification method based on image recognition according to any one of claims 1 to 7, characterized in that: The system comprises: a recognition module, configured to acquire multiple frames of continuous time series images containing lysosomes, establish a recognition model, and output a predicted probability map of the presence of lysosomes in the time series images based on the recognition model; an overlap determination module, configured to identify a peak position in the predicted probability map, define the peak position as a candidate position of a lysosome in the time series image, determine whether there is an overlap at the candidate position, and if so, define the candidate position as an overlap region; a segmentation module, configured to obtain a light field image at the overlapping region, segment the overlapping region into a plurality of independent regions based on the light field image, and obtain the three-dimensional coordinates of a lysosome in each of the independent regions; The classification module is used to identify the same lysosome in multiple frames of continuous time series images based on the three-dimensional coordinates, generate the dynamic motion trajectory of the same lysosome, establish a classification model, obtain a motion trajectory data set with primary lysosomes and secondary lysosomes respectively labeled, use the motion trajectory data set as training data for the classification model, and identify the type of the lysosome based on the classification model; Step S1 outputs a predicted probability map of the existence of lysosomes in the time series images, including the following steps: obtaining a first cell image labeled with primary lysosomes or secondary lysosomes, establishing a recognition model based on a neural network model, training the recognition model based on the first cell image, and The time series image is input into the trained recognition model, and a predicted probability map is output, wherein the predicted probability map includes the predicted probability of the existence of lysosomes at each pixel in the time series image, and the position information and size information of each lysosome in the first cell image are identified and annotated to obtain a second cell image. Based on the second cell image, a true score map of the existence of the lysosomes is calculated, and a deviation value between the predicted probability map and the true score map is calculated. If the deviation value is greater than a first threshold, the parameters of the recognition model are adjusted, and the recognition model obtains a new predicted probability map based on the adjusted parameters, and this operation is repeated until the deviation value is less than or equal to the first threshold.

Images