A CLIP-based open-set organoid localization recognition method

Abstract

The application discloses a kind of open set organoid positioning identification methods based on CLIP, comprising the following steps: S1, obtain organoid data, and it is preprocessed;S2, based on CLIP model builds end-to-end positioning identification model;S3, the organoid data after pre-processing is sent into positioning identification model and inferences;S4, the inference result is restored to original image, and then non-maximum suppression processing is carried out, to realize the positioning identification of organoid;The CLIP model is introduced into the organoid positioning identification task in the application, so that the original Faster-RCNN positioning identification model retains the advantages of end-to-end, high precision, etc. Under the premise, it does not need artificial specific design in training and inference stage, can consider model performance, improve algorithm efficiency, can guarantee the positioning identification performance of known category data at the same time, also has certain positioning identification ability to unknown category data, that is, it is helpful to the positioning identification of new type category, improves the generalization ability of model.

Description

Technical Field

[0001] This invention relates to the field of computer processing technology, and more specifically to an open set organoid localization and recognition method based on CLIP. Background Technology

[0002] Organoids are 3D cell cultures that are highly similar to their source tissues and organs, and contain some of the key characteristics of the source tissues and organs. In vitro cultured organoid systems are organ-specific cell types capable of self-renewing stem cell populations and differentiating into multiple organs. They possess similar spatial organization to their corresponding organs and can reproduce some of the functions of the corresponding organs, thus providing a highly physiologically relevant system. Research on organoids is an important research topic in the fields of biopharmaceuticals and stem cells.

[0003] Currently, methods for localizing and recognizing organoid images can be mainly divided into several categories: 1. Using a sliding window approach, the image is sliced ​​into small pieces according to a set pattern and fed into a classification algorithm model for image recognition. Commonly used algorithms include ResNet. This method is suitable for image data with large pixel counts. Its main drawback is that different image slicing methods are required for different target sizes, and the localization is often inaccurate and may miss some targets. 2. Integrating localization and recognition into a single model, using an end-to-end approach to locate and recognize targets within the image. This allows the model to recognize targets of any size appearing at any location in the image. Commonly used algorithms include SSD and Faster-RCNN. This type of algorithm is also one of the more commonly used solutions, but it cannot handle excessively large image data well. Furthermore, in addition to the advantages and disadvantages mentioned above, current methods are only applicable to data that has already appeared in the training data and cannot effectively locate and recognize data that has not appeared in the training data. Summary of the Invention

[0004] The purpose of this invention is to provide an open set organoid localization and identification method based on CLIP.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A CLIP-based open-set organoid localization and recognition method includes the following steps:

[0007] S1. Obtain organoid data and preprocess it;

[0008] S2. Build an end-to-end positioning and recognition model based on the CLIP model;

[0009] S3. Input the preprocessed organoid data into the localization and recognition model for inference;

[0010] S4. Restore the inference results to the original image, and then perform non-maximum suppression processing to achieve organoid localization and recognition.

[0011] Preferably, the specific process of step S1 is as follows:

[0012] S11. Using limbal stem cells in organoids as representative data, acquire image data of stem cells in organoids. These stem cells include three categories: holoclone, meroclone, and paraclone. Define the data from the holoclone and meroclone categories as the basic data class, denoted as D. B Define the data of category paraclone as a new data category, denoted as D. N .

[0013] S12. Define the entire original image data as D. Slice the entire image D into image patch data of size 896x896 pixels. The slicing method is as follows: determine the location of a labeled cell data, and on the premise that this cell is included, randomly generate an image patch of size 896x896. Record the relative position coordinates of the remaining cells completely contained in this image patch. This generates a set of training image data D. train Training data D is required. train It contains only D B Category data.

[0014] Preferably, the localization and recognition model in step S2 consists of a detection module and a distillation module; the detection module consists of a backbone network, several convolutional layers and a fully connected layer; the distillation module shares the same backbone network with the detection module, and at the same time, several new convolutional layers and a fully connected layer are connected to the backbone network.

[0015] Preferably, the data processing procedure for the positioning and recognition model in step S2 is as follows:

[0016] S21. For image data, the image data D generated in step S1... train The image data D is fed into the distillation module as training and labeling data to obtain image features R1. train The image data D is then sent to the detection module to obtain image features R2; train The image feature encoding module of the pre-trained CLIP model is used to encode the CLIP image feature I, thus obtaining CLIP image feature I.

[0017] S22. For text data, replace the three labels holoclone, meroclone and paraclone with their corresponding specific text descriptions, and encode them through the text feature encoding module of the pre-trained CLIP model to obtain CLIP text features B. During training, only the text features of the two labels holoclone and meroclone will be used, while during inference, the text features of the three labels mentioned above will be used.

[0018] S23. Distillation optimization is performed on image vector features R1 and image vector features I using the L1 Loss loss function, which is:

[0019]

[0020] Where N is the dimension of the feature vector generated by the model, which is 512, and the value of j ranges from 1 to N;

[0021] S24. Compare the image feature R2 with the text feature B using cosine similarity to obtain the similarity y', and then use the cross-entropy function for classification optimization; where the cosine similarity is denoted as y', the true one-hot label data is y, and the loss function is:

[0022]

[0023] Where N is the dimension of the feature vector generated by the model, which is 512, and the value of j ranges from 1 to N.

[0024] Preferably, the specific process of step S3 is as follows:

[0025] S31. Slice the entire original image data D into 896x896 pixel slices. The slicing method is as follows: slice the image from left to right and from top to bottom into 896x896 pixel slices, with a 448 pixel overlap between every two images on the left and right or top and bottom. If there are not enough pixels on the right or bottom, cut an 896x896 pixel image block to the left or top as a slice, using the right or bottom as the ending edge. Record the position of all slices on the original image to generate a set of inference image data D. test D test The data contains D B Category data and D N Category data;

[0026] S32, D test The data is fed into the positioning and recognition model, and the distillation module infers the reasoning result P. R1 The reasoning result P is obtained by the detection module. R2 ;

[0027] S33. The reasoning result P is obtained by reasoning from the distillation module. R1 The reasoning result P is obtained by reasoning with the detection module. R2 The integration will be carried out in the following manner:

[0028]

[0029] Where λ takes the value 2 / 3; for P R1 and P R2 Let j be the index of the category label in the predicted score; j∈D N The label corresponding to the current index belongs to D. N Category, j∈D B The label corresponding to the current index belongs to D. B category.

[0030] Preferably, the specific process of step S4 is as follows:

[0031] S41. Based on the position of the recorded slice on the original image D, restore the inference result to the original image D, and then perform NMS non-maximum suppression processing to obtain the final inference result.

[0032] S42. Visualize the reasoning results on the original image to ultimately achieve the localization and recognition of organoids.

[0033] After adopting the above technical solution, the present invention has the following beneficial effects: The present invention introduces the CLIP model into the organoid localization and recognition task, enabling the original Faster-RCNN localization and recognition model to retain its advantages such as end-to-end processing and high accuracy, while also possessing a certain localization and recognition capability for categories not appearing in the training data, thus improving the model's generalization ability. Furthermore, the localization and recognition model of the present invention has an end-to-end structure, eliminating the need for manual specialization design during the training and inference stages. This balances model performance and improves algorithm efficiency, providing a certain localization and recognition capability for unknown categories without compromising localization and recognition performance for known category data, thus facilitating the localization and recognition of novel categories. Attached Figure Description

[0034] Figure 1 This is a flowchart of the present invention;

[0035] Figure 2 This is a schematic diagram of the positioning and recognition model of the present invention;

[0036] Figure 3 This is a flowchart of the data processing for the positioning and identification model of the present invention;

[0037] Figure 4 This is a visualization of the results of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] Example

[0040] like Figures 1 to 4 As shown, an open-set organoid localization and recognition method based on CLIP includes the following steps:

[0041] S1. Obtain organoid data and preprocess it;

[0042] The specific process of step S1 is as follows:

[0043] S11. Using limbal stem cells in organoids as representative data, acquire image data of stem cells in organoids. These stem cells include three categories: holoclone, meroclone, and paraclone. Define the data from the holoclone and meroclone categories as the basic data class, denoted as D. B Define the data of category paraclone as a new data category, denoted as D. N During the training process, data D is used. B Train and fine-tune the model, and use the data D during the inference phase. B and data D N Both are used as test data to verify the algorithm's performance, and it is ensured that the training data and test data have no overlap;

[0044] S12. Define the entire original image data as D. Slice the entire image D into image patch data of size 896x896 pixels. The slicing method is as follows: determine the location of a labeled cell data, and on the premise that this cell is included, randomly generate an image patch of size 896x896. Record the relative position coordinates of the remaining cells completely contained in this image patch. This generates a set of training image data D. train Training data D is required. train It contains only D B Category data;

[0045] S2. Build an end-to-end positioning and recognition model based on the CLIP model;

[0046] The localization and recognition model described in step S2 consists of a detection module and a distillation module; the detection module consists of a backbone network, several convolutional layers and a fully connected layer; the distillation module shares the same backbone network with the detection module, and at the same time, several new convolutional layers and a fully connected layer are connected to the backbone network.

[0047] The data processing procedure for the positioning and recognition model in step S2 is as follows:

[0048] S21. For image data, the image data D generated in step S1... train The image data D is fed into the distillation module as training labeled data to obtain image features R1. train The image data D is then sent to the detection module to obtain image features R2; train The image feature encoding module of the pre-trained CLIP model is used to encode the CLIP image feature I, thus obtaining CLIP image feature I.

[0049] S22. For text data, replace the three labels holoclone, meroclone and paraclone with their corresponding specific text descriptions, and encode them through the text feature encoding module of the pre-trained CLIP model to obtain CLIP text features B. During training, only the text features of the two labels holoclone and meroclone will be used, while during inference, the text features of the three labels mentioned above will be used.

[0050] S23. Distillation optimization is performed on image features R1 and image features I using the L1 Loss loss function, which is:

[0051]

[0052] Where N is the dimension of the feature vector generated by the model, which is 512, and the value of j ranges from 1 to N;

[0053] S24. Compare the image feature R2 with the text feature B using cosine similarity to obtain the similarity y', and then use the cross-entropy function for classification optimization; where the cosine similarity is denoted as y', the true one-hot label data is y, and the loss function is:

[0054]

[0055] Where N is the dimension of the feature vector generated by the model, which is 512, and the value of j ranges from 1 to N;

[0056] S3. Input the preprocessed organoid data into the localization and recognition model for inference;

[0057] The specific process of step S3 is as follows:

[0058] S31. Slice the entire original image data D into 896x896 pixel slices. The slicing method is as follows: slice the image from left to right and from top to bottom into 896x896 pixel slices, with a 448 pixel overlap between every two images on the left and right or top and bottom. If there are not enough pixels on the right or bottom, cut an 896x896 pixel image block to the left or top as a slice, using the right or bottom as the ending edge. Record the position of all slices on the original image to generate a set of inference image data D. test D test The data contains D B Category data and D N Data by category.

[0059] S32, D test The data is fed into the positioning and recognition model, and the distillation module infers the reasoning result P. R1 The reasoning result P is obtained by the detection module. R2 ;

[0060] S33. The reasoning result P is obtained by reasoning from the distillation module. R1 The reasoning result P is obtained by reasoning with the detection module. R2 The integration will be carried out in the following manner:

[0061]

[0062] Where λ takes the value 2 / 3; for P R1 and P R2 Let j be the index of the category label in the predicted score; j∈D N The label corresponding to the current index belongs to D. N Category, j∈D B The label corresponding to the current index belongs to D. B category;

[0063] S4. Restore the inference results to the original image, and then perform NMS non-maximum suppression processing to achieve organoid localization and recognition.

[0064] The specific process of step S4 is as follows:

[0065] S41. Based on the position of the recorded slice on the original image D, restore the inference result to the original image D, and then perform non-maximum suppression processing to filter out redundant results and obtain the final inference result.

[0066] S42. Visualize the reasoning results on the original image to ultimately achieve the localization and recognition of organoids.

[0067] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A CLIP-based open-set organoid localization recognition method, characterized in that, The method comprises the following steps: S1, obtaining organoid data and pre-processing the same; The specific process of step S1 is as follows: S11, taking the corneal limbal stem cells in the organoids as representative data, obtaining stem cell image data in the organoids, the stem cells including three types of holoclone, meroclone and paraclone, defining two types of data of holoclone and meroclone as basic class data, denoted as D B , and defining the data of paraclone as new class data, denoted as D N . S12, define the whole original image data as D, slice the whole image D according to the size of 896x896 pixels to obtain image block data; wherein the slicing manner is: determining a certain labeled cell data position, taking the inclusion of the cell as a premise, randomly generating a picture block with a size of 896x896, and recording the relative position coordinates of the remaining cells completely contained in the picture block, thereby generating a set of training image data D train , requiring that the training data D train only contains D B category data; S2, building an end-to-end positioning recognition model based on a CLIP model; The positioning recognition model in step S2 is composed of a detection module and a distillation module; the detection module is composed of a backbone network, a plurality of convolutional layers and a fully connected layer; the distillation module shares the same backbone network with the detection module, and additionally connects a plurality of new convolutional layers and a fully connected layer on the backbone network; The data processing process of the positioning recognition model in step S2 is as follows: S21, for image data, the image data D generated in step S1 is processed train As the training label data is input into the distillation module to obtain the image feature R1, the image data D train is input into the detection module to obtain the image feature R2; then the image data D train is encoded by the image feature encoding module of the pre-trained CLIP model to obtain the CLIP image feature I; S22, for text data, replace the holoclone, meroclone and paraclone three types of labels with their corresponding specific text descriptions respectively, encode through the text feature encoding module of the pre-trained CLIP model to obtain CLIP text features B; only the text features of the holoclone and meroclone two types of labels will be used in the training process, and the text features of the above three types of labels will be used in the inference process; S23, distill and optimize the image features R1 and the image features I using an L1Loss loss function, and the loss function is as follows: Wherein, N is the dimension of the feature vector generated by the model, which is 512, and the value range of j is 1 to N; S24, compare the cosine similarity of the image features R2 and the text features B to obtain the similarity y', and then use the cross-entropy function for classification optimization; wherein, let the cosine similarity be y', the true value one-hot label data be y, and the loss function be: S3, input the pre-processed organoid data into the positioning recognition model for inference; S4, restore the inference result to the original image and perform non-maximum suppression processing to realize the positioning recognition of the organoid.

2. The CLIP-based open-set organoid localization and identification method of claim 1, wherein, The specific process of step S3 is as follows: S31, slice the whole original image data D according to the size of 896x896 pixels, the slicing mode is: from left to right and from top to bottom, slice the image according to the size of 896x896 pixels, there are 448 pixel size overlaps between every two pictures on the left or right or top and bottom, if the right or bottom pixels are insufficient, the right or bottom is the end edge, the image block with the size of 896x896 is intercepted to the left or top as a slice, and the position of all slices on the original image is recorded to generate a set of inference image data D test , test D B The data in the data D N category and D N category data; S32, the D test data into the positioning recognition model, and the inference result P is obtained by inference of the distillation module R1 , and the inference result P is obtained by inference of the detection module R2 ; S33, the distillation module reasoning obtains a reasoning result P R1 with the detection module reasoning obtains a reasoning result P R2 fusion, and the fusion manner is: where λ takes 2 / 3; for P R1 and P R2 , let j be the index of the class label corresponding position in the prediction result score; j∈D N denotes that the label corresponding to the current index belongs to the D N class, j∈D B denotes that the label corresponding to the current index belongs to the D B class.

3. The CLIP-based open-set organoid localization and identification method of claim 2, wherein, The specific process of step S4 is as follows: S41, according to the recorded position of the slice on the original image D, restore the inference result to the original image D, and then perform non-maximum suppression processing to filter out redundant results and obtain the final inference result; S42, visualize the inference result on the original image to finally realize the positioning recognition of the organoid.

Images