High-throughput drug screening method and system based on large-field-of-view imaging and machine learning

Abstract

According to the high-throughput drug screening method and system based on large-view-field imaging and machine learning, data acquisition of high-throughput organs or tumor balls is achieved through a large-view-field biological imaging system, the data acquisition process is accelerated, and the integrity of original image information is guaranteed; a two-stage image processing means that a lightweight target detection model is connected with a semantic segmentation model is adopted, so that the calculation parameter quantity is reduced, heterogeneity interference is removed, and the image information integrity is ensured while the accurate acquisition of the individual image contour of the organoid or the tumor ball is realized; for organoid image acquisition, a Z-axis scanning strategy is combined with a screening strategy based on IOU and a definition function, so that information acquisition integrity and subsequent organoid activity prediction precision are ensured; in combination with a machine learning model based on an ensemble learning principle for a regression task, the activity prediction of the bright field image of the organoid or tumor sphere is realized, and the influence of a traditional method on the activity of a culture is avoided.

Description

Technical Field

[0001] This invention relates to the field of biomedical technology, and in particular to a high-throughput drug screening method and system based on large field-of-view imaging and machine learning. Background Technology

[0002] In drug screening experiments and drug sensitivity testing for precision medicine in clinical settings, assessing drug efficacy is a crucial step in determining treatment effectiveness. Drug screening is often a lengthy, complex, and expensive process. Furthermore, due to individual biological differences, different patients often require rapid and targeted development of optimal treatment plans. Therefore, achieving high-throughput drug screening has become a research hotspot in the fields of precision medicine and biomedicine in recent years.

[0003] Traditional drug screening often relies on two-dimensional adherent cell models and animal models. The former are readily available, highly stable, and inexpensive, but their cell-to-cell interactions are weak, making it difficult to create physiological structures that closely resemble real organs. The latter can reflect the in vivo environment to some extent, but suffers from high cost, long processing times, and limitations due to species differences. In contrast, three-dimensional organ models such as tumor spheres and organoids, because they more closely resemble real tissue environments, have become important tools for drug screening.

[0004] Currently, laboratories often evaluate the activity of organoids and tumor spheres by adding ATP reagent to the culture to detect luminescence signals or by using live-dead staining to observe the fluorescence ratio. However, these methods can directly or indirectly damage the culture, rendering it unusable for subsequent experiments and resulting in a waste of resources.

[0005] For example, Chinese patent ZL202210110925.6 discloses a method and apparatus for detecting artificial tissues and organoids. Chinese patent ZL202210868607.6 discloses an artificial intelligence-based organoid ATP analysis method and system. Chinese patent ZL202410707272.9 discloses an organoid ATP analysis method and system based on multi-instance learning. Chinese patent application CN202410712816.0 discloses an organoid ATP prediction method based on a local-global graph neural network. A paper with DOI 10.1364 / BOE.486666 develops a method for label-free, continuous tracking imaging and quantitative analysis of drug efficacy using organoids. A master's thesis published in 2023, titled "Research on the Application of Organoid Image Recognition Based on Artificial Intelligence," designed a machine learning-based organoid classification method based on the morphological differences of different types of organoids.

[0006] The scheme disclosed in Chinese Patent ZL202210110925.6 directly segments the acquired image, which may lead to unnecessary waste of computational resources. Furthermore, it only extracts morphological indicators from the segmentation results, ignoring potentially valuable information such as daytime and texture indicators. Secondly, this patent and the paper with DOI 10.1364 / BOE.486666 fail to demonstrate that the evaluation indicators obtained through principal component analysis have universality or are replaceable ATP indicators. The method disclosed in Chinese Patent ZL202210868607.6, which uses small field-of-view stitching to obtain large field-of-view images, cannot fundamentally solve the low-throughput problem. This patent, along with the methods provided in Chinese Patent ZL202410707272.9 and Chinese Patent Application CN202410712816.0, also directly inputs the large field-of-view image obtained through stitching into a convolutional neural network, potentially making it difficult to balance prediction accuracy and computational load. The master's thesis, "Research on the Application of Artificial Intelligence in Organoid Image Recognition," only conducted a binary classification experiment to determine whether organoids were affected by drugs, and failed to demonstrate organoid activity through continuous quantitative data.

[0007] In summary, existing technologies are all performed within a small field of view, which cannot achieve high throughput. Furthermore, the deep learning processes all require the input of a complete image, which may result in significant computational overhead.

[0008] Therefore, it is necessary to provide a new approach to solve the aforementioned technical problems. Summary of the Invention

[0009] To achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide a high-throughput drug screening method based on large field-of-view imaging and machine learning, comprising the following steps: Images of each well of a drug-sensitive plate acquired using a large field-of-view imaging system are used to construct a target detection dataset; wherein, the drug-sensitive plate contains drug-treated tumor spheres or organoids. Obtain the actual ATP cell viability in each well of the test; Label each image with in-focus and out-of-focus objects, and crop out the in-focus single objects to construct a semantic segmentation dataset; The image is input into the object detection model for training, so that the model can locate and classify the in-focus target objects and the out-of-focus objects. The semantic segmentation dataset is input into the semantic segmentation model for image segmentation to accurately obtain object contours. Extract the spatiotemporal features of all objects from the image within each hole and calculate the average value; Using the information within each well as a single sample, the average of multiple features within the well as multiple features, and the actual ATP cell viability as the label value, a regression machine learning network model is trained. The trained model is then used to predict the viability of bright-field images.

[0010] Furthermore, the step of acquiring images within each well of the drug-sensitive plate obtained under a large field-of-view imaging system and constructing a target detection dataset further includes: When constructing the organoid dataset, a Z-axis pore-by-pore scan was performed, and multiple images were acquired for each pore.

[0011] Furthermore, the step of labeling in-focus and out-of-focus objects on each image, and cropping out the in-focus individual objects for constructing the semantic segmentation dataset includes: For each hole, perform focus-normal object detection on all images, and apply IOU matching and filtering strategies to images of different planes within the same hole: If the Intersection over Union (IOU) value of the same organoid is greater than a threshold in all images, it is considered a duplicate object. For the same object's detection bounding box region, only the one with the highest sharpness evaluation function is retained. All non-duplicate organoids are cropped for subsequent segmentation and regression processes. The IOU calculation formula is as follows: .

[0012] Furthermore, the spatiotemporal features include morphological features, texture features, and daytime features.

[0013] Furthermore, the morphological features include area, perimeter, roundness, elongation, eccentricity, central moment, and concavity / convexity; The texture features include a gray-level co-occurrence matrix, a gray-level run-length matrix, and a gray-level region size matrix; The daytime feature is the ratio of the morphological feature and the texture feature between every two observation times.

[0014] Furthermore, it also includes steps for feature selection and PCA to remove redundancy and reduce dimensionality of features: Spearman correlation analysis was performed between each feature and ATP test results, and features showing a strong correlation with organoid or tumor sphere viability were selected for further analysis. The selected features are standardized using the Z-score method. The standardized features are then subjected to Pearson correlation coefficient calculation. Features with high correlation are extracted and subjected to PCA dimensionality reduction. The extracted highly correlated feature matrix A is an array with m rows and l columns, where m is the total number of culture wells used in the multi-well antimicrobial susceptibility plate, and l is the number of features. It is the value of the j-th feature in the i-th culture well; The weights of the culture characteristic parameters are calculated through eigenvalue decomposition, and the culture characteristic parameters are normalized according to their weight coefficients. The covariance between the culture characteristic parameters is calculated to construct a covariance matrix B with l rows and l columns. In matrix B, cov() represents the covariance. Represents the j-th feature parameter. ; The eigenvalues ​​and eigenvectors of the covariance matrix are calculated by eigenvalue decomposition, forming a sequence of eigenvalues ​​from maximum to minimum. The eigenvectors corresponding to the sorted eigenvalues ​​are used as weighting coefficients for organ-like feature parameters. The principal components corresponding to the eigenvalues ​​are retained from largest to smallest until the sum of the eigenvalues ​​of all principal components is reached. Greater than the threshold The feature extraction process is completed by combining these principal components with the original relevant features to form a new feature series; where the variance proportion of each principal component is... for: in, is the i-th feature value, and l is the number of dimensions of the feature; Cumulative variance explained proportion: Among them, the proportion of cumulative variance explained reaches a threshold. k principal components at time.

[0015] The second objective of this invention is to provide a high-throughput drug screening method based on large field-of-view imaging and machine learning. The method includes the following steps: Acquire images of the inside of the drug sensitivity test plate wells and natural language commands input by the user; The large language model performs inference based on natural language instructions and calls various functional modules as needed to generate the corresponding response.

[0016] A third objective of this invention is to provide a high-throughput drug screening system based on large field-of-view imaging and machine learning, employing the aforementioned methods, including a large field-of-view imaging system, a data processing module, and a human-computer interaction module based on a large language model; wherein, The optical lens of the large field-of-view imaging system includes a primary mirror, a secondary mirror, a third mirror, and a folding mirror. The primary mirror and the third mirror are freeform surfaces, the secondary mirror is a quadratic surface, and the folding mirror is a plane mirror. Light from the object side enters the optical lens and reaches the primary mirror. After being reflected by the primary mirror, it forms a first reflected light. The secondary mirror is placed in the reflected light path of the primary mirror and reflects the first reflected light from the primary mirror to form a second reflected light. The third mirror is placed in the reflected light path of the secondary mirror and reflects the second reflected light from the secondary mirror to form a third reflected light. The folding mirror is placed in the reflected light path of the third mirror and reflects the third reflected light from the third mirror to form a fourth reflected light. The fourth reflected light forms an image at the image plane. The data processing module is used to acquire images of each well of an antimicrobial susceptibility plate collected under a large field-of-view imaging system, and construct a target detection dataset. The antimicrobial susceptibility plate contains drug-treated tumor spheres or organoids. The module acquires the actual ATP cell viability in each well. It labels each image with focused and unfocused objects, and crops out single focused objects to construct a semantic segmentation dataset. The images are input into a target detection model for training, enabling the model to locate and classify focused and unfocused objects. The semantic segmentation dataset is input into the semantic segmentation model for image segmentation to accurately obtain object contours. All spatiotemporal features of objects are extracted from the images within each well, and their average values ​​are calculated. Using the information within each well as a single sample, the average value of multiple features within the well as multiple features, and the actual ATP cell viability as a label value, a regression machine learning network model is trained. The trained model is used to predict the viability of bright-field images. The human-computer interaction module based on the large language model is used to acquire the image inside the well of the drug sensitivity plate and natural language commands input by the user, perform inference based on the natural language commands, call various functional modules as needed, and finally generate the corresponding response.

[0017] A fourth objective of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0018] A fifth objective of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention enables high-throughput data acquisition of organoids or tumor spheres through a large field-of-view bioimaging system, accelerating the data acquisition process and ensuring the integrity of the original image information.

[0020] This invention employs a two-stage image processing approach that combines a lightweight target detection model with a semantic segmentation model. This approach achieves accurate acquisition of individual image contours of organoids or tumor spheres while reducing the number of computational parameters, removing heterogeneous interference, and ensuring the integrity of image information.

[0021] This invention addresses organoid image acquisition by employing a Z-axis scanning strategy combined with a screening strategy based on IOU and a sharpness function to ensure the integrity of information acquisition and the accuracy of subsequent organoid viability prediction.

[0022] This invention combines a machine learning model based on ensemble learning principles for regression tasks to predict the activity of organoids or tumor spheres from bright-field images, avoiding the impact of traditional methods on culture activity. This process is applicable not only to organoids but also to tumor sphere activity evaluation. In experiments with specific requirements for experimental duration and accuracy, tumor spheres may be a superior organ model.

[0023] Compared to one-stop bright-field image-based culture activity prediction, this invention employs a three-stage feature extraction and activity prediction method based on target detection, semantic segmentation, and regression models. This method can obtain more clear and promising intermediate layer information, such as the number and growth distribution of different types of organoids or tumor spheres. Furthermore, this method provides more possibilities for expanding the data analysis capabilities of subsequent large language models.

[0024] This invention integrates large language model technology into the user-end software, enabling doctors or other operators without programming experience to easily complete image analysis through voice or text commands. Furthermore, this technology enhances user flexibility, for example, allowing users to customize the display method and style of statistical visualization results.

[0025] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of a high-throughput drug screening method based on large field-of-view imaging and machine learning. Figure 1 ; Figure 2 This is a schematic diagram of the imaging principle of an off-axis three-mirror system. Figure 3 This is a schematic diagram of an off-axis three-mirror imaging system; Figure 4 A flowchart for high-throughput drug screening; Figure 5 Large field-of-view organoid diagram; Figure 6 A large field-of-view tumor sphere image; Figure 7 A full-field tumor sphere image; Figure 8 Flowchart for constructing 3D organoids; Figure 9 Flowchart for constructing 3D tumor spheres using the liquid overlay method; Figure 10 The process and results of organoid testing; Figure 11 The process and results of organoid segmentation; Figure 12 This is a schematic diagram illustrating the principle of organoid viability prediction based on machine learning. Figure 13 This is a flowchart of a high-throughput drug screening method based on large field-of-view imaging and machine learning. Figure 2 ; Figure 14 This is a flowchart of image analysis based on a large language model. Figure 15 A schematic diagram of a high-throughput drug screening system based on large field-of-view imaging and machine learning; Figure 16 This is a schematic diagram of a computer device. Figure 17 This is a schematic diagram of a computer-readable storage medium. Detailed Implementation

[0027] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0028] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0029] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0031] To address the demands of high-throughput drug screening, this invention transfers the traditional drug screening process to an off-axis, large-field-of-view (OLT) reflex mirror instrument in the laboratory. To address the issue of large image information volume in large-field-of-view images, a lightweight two-stage image processing workflow is designed to extract features from target cultures. Utilizing the extracted spatiotemporal features (feature changes over time and feature distribution in space), and using the luminescence values ​​extracted by ATP luminescence detection as the standard activity evaluation method as the true label values, a lightweight regression machine learning network is trained to accurately predict the viability of bright-field cultures. Furthermore, a large language model is incorporated to provide efficient support for human-computer interaction and data analysis. Compared to organoid-based drug screening workflows, tumor sphere construction offers advantages such as shorter cycle times and lower costs. The workflow in this method is applicable to both organoid-based and tumor sphere-based drug screening experiments.

[0032] Example 1 A high-throughput drug screening method based on large field-of-view imaging and machine learning, such as Figure 1 , Figure 4 As shown, it includes the following steps: S100. Acquire images of each well of the drug-sensitive plate under a large field-of-view imaging system and construct a target detection dataset; wherein, the drug-sensitive plate contains drug-treated tumor spheres or organoids. This embodiment is based on a large field-of-view imaging system with a field of view of 50×10mm. First, organoids or tumor spheres are placed into a 96-well drug sensitivity plate and cultured to a suitable size of approximately 50µm. Next, the 96-well plate is removed, and different concentrations, types, or combinations of drugs are added. A control group and at least three parallel wells are included. For drug concentration gradient screening experiments, concentrations of 0ug / ml, 0.01ug / ml, 0.1ug / ml, 1ug / ml, 5ug / ml, 10ug / ml, and 100ug / ml are commonly used. The drug-treated tumor spheres or organoids are cultured for 3 days to allow for drug action. If multiple groups of organoids or tumor spheres are found to be inactivated during drug action, they can be removed for subsequent experiments. After removing the culture plate, images of each well are acquired using the large field-of-view bioimaging system to construct a target detection dataset. The target detection dataset should contain at least 5000 images, including at least 10 individual patients, to ensure broad coverage of individual variability.

[0033] The organoid construction process is as follows: Figure 8As shown. First, the primary tissues and organs are cleaned and minced. After mincing, the tissue and digestion solution are mixed using a pipette tip and digested for 20 minutes. During digestion, the tissue is checked periodically to observe the digestion effect and determine whether it has reached the standard of complete digestion. For difficult-to-digest samples, the digestion time can be appropriately increased, generally not exceeding 1 hour. After digestion, a sieve is placed on a new 15ml centrifuge tube, and liquid is drawn from the center of the sieve to sieve the cells. The cell density is observed under a microscope. Finally, the density is diluted / concentrated to the optimal level. The matrix gel is mixed with the cell suspension and then seeded into culture plates. The optimal concentration and seeding amount need to be determined according to the culture plates.

[0034] The process of constructing tumor spheres is as follows Figure 9 As shown, tumor cells are first cultured to the logarithmic growth phase, then digested and prepared into a single-cell suspension. After adjusting the cell density to a suitable level for inoculation, the cells are seeded onto an ultra-low adsorption plate and placed in a 37°C, 5% CO2 incubator. After 1–3 days, cells can be observed to gradually aggregate and begin to form tumor spheres. The optimal inoculation concentration and amount need to be determined based on the culture vessel.

[0035] The optical principle of an off-axis three-mirror large field-of-view imaging system is as follows: Figure 2 As shown. Off-axis reflective systems offer advantages such as no central obstruction, a large field of view, and no chromatic aberration. Therefore, they are widely used in imaging applications such as infrared imaging, space cameras, and remote sensing. The mirrors used in off-axis reflective systems are commonly spherical, conical, aspherical, and freeform surfaces.

[0036] The optical path of the laboratory large field-of-view imaging system is as follows: Figure 3 As shown, it includes a primary mirror 1, a secondary mirror 2, a third mirror 3, and a folding mirror 4. The primary and third mirrors are freeform surfaces, the secondary mirror is a quadric surface, and the folding mirror is a plane mirror. Light rays from the object side entering the optical lens first reach the primary mirror and are reflected to form the first reflected light. The secondary mirror is positioned in the reflection path of the primary mirror, reflecting the first reflected light to form the second reflected light. The third mirror is positioned in the reflection path of the secondary mirror, reflecting the second reflected light to form the third reflected light. The folding mirror is in the reflection path of the third mirror, reflecting the third reflected light to form the fourth reflected light. The fourth reflected light will form an image at the image plane. The reflectors are made of microcrystalline material, a common material for reflective optical-mechanical structures, which has advantages such as low cost, easy processing, short cycle time, and mature processing technology.

[0037] Compared to conventional transmission imaging systems, imaging systems based on the off-axis three-mirror principle offer advantages such as a large field of view, wide spectral range, low distortion, and small size. After assembly and testing, the system achieved a resolution of 1.5µm, a single image field of view of up to 50×10mm, and a magnification of up to 2x. Its field of view is approximately 10 times larger than that of typical commercial microscope cameras. For example, the Ti-2U Nikon microscope's image plane can cover a maximum field of view of 25mm in diameter, and due to limitations in camera target size, even with a 10x objective lens, the acquired image typically covers less than 2.5mm of field of view.

[0038] Figure 7 The instrument displays full-field tumor sphere images, with a single image covering more than six wells. Compared to traditional microscopes, which cannot achieve single-well coverage in a single image, experiments conducted on this instrument can achieve a significant increase in throughput.

[0039] S200, Obtain the actual ATP cell viability in each well of the test; S300. Label the in-focus objects and out-of-focus objects on each image, and crop out the in-focus single objects to construct a semantic segmentation dataset. In some embodiments, the step of acquiring images of each well of the drug-sensitive plate collected under a large field-of-view imaging system and constructing a target detection dataset further includes: Since organoids grow in three-dimensional matrix droplets, a Z-axis well-by-well scan is performed during the construction of the organoid dataset, and multiple images are acquired for each well.

[0040] To ensure complete scanning of the adhesive droplets, a scanning step size of 0.1 mm and a scanning travel distance of 2 mm were used, resulting in 20 images per well. Further, the step of labeling in-focus and out-of-focus objects on each image, and cropping out the in-focus single objects for constructing the semantic segmentation dataset, includes: For each hole, perform focus-normal object detection on all images, and apply IOU matching and filtering strategies to images of different planes within the same hole: If the Intersection over Union (IOU) value of the same organoid is greater than a threshold (e.g., the threshold is set to 0.8) in all images, it is considered a duplicate object. For the same object's detection bounding box region, only the one with the highest index, i.e., the clearest object, is retained according to the sharpness evaluation function. All non-duplicate organoids are cropped for subsequent segmentation and regression processes. The IOU calculation formula is as follows: .

[0041] Sharpness evaluation functions can include Brenner gradient function, contrast, Laplacian variance, Tenengrad index, etc.

[0042] At this point, ≥20,000 individual object images can be collected as a semantic segmentation dataset.

[0043] S400. Input the image into the target detection model for training, so that the model can locate and classify the in-focus target objects and out-of-focus objects. In this embodiment, the image is input to the target detection model for training, so that the model can automatically locate and classify the normally focused target objects and the out-of-focus objects.

[0044] The process for detecting organoids or tumor spheres is as follows: Figure 10 As shown, the dataset is first acquired under a large field-of-view system. Since organoids exhibit three-dimensional growth within the droplet, data acquisition requires traversing the entire droplet with a certain step size to avoid losing necessary information. Next, the image is cropped out for each hole individually, and each hole image is labeled using LabelMe software. Labeled objects are categorized into two types: in-focus and out-of-focus. Out-of-focus objects in the image need to be ignored in subsequent semantic segmentation. After acquiring and labeling the dataset, it is divided into training and validation sets. The training set data is enhanced using methods such as rotation, cropping, and mosaicking, and then input into the object detection model for training. Taking the YOLO object detection model as an example, the model structure includes a backbone network, a neck network, and a detection head. The backbone network is typically a deep convolutional neural network used to extract features of target objects in the image. The deep feature map after feature extraction by the backbone network has rich semantic information. The neck network is used for multi-scale feature fusion and enhancement, ensuring that features of both large and small targets are completely collected. The detection head is the output of the object detection network, providing information such as the predicted object location, prediction confidence, and predicted category. Furthermore, object detection networks can also employ Transformer-based models such as DETR, RTDETR, Dfine, and DEIM.

[0045] S500: Input the semantic segmentation dataset into the semantic segmentation model to perform image segmentation and accurately obtain the object contour; Organoid or tumor sphere segmentation process as follows Figure 11 As shown, firstly, objects labeled as being in focus in the object detection dataset are cropped out. Then, the object outlines are drawn using the LabelMe software to create a semantic segmentation dataset. LabelMe integrates the SAM segmentation tool for auxiliary annotation. After dividing the dataset into training and validation sets, the training set, containing a large proportion of data, is input into the semantic segmentation model for training to determine the model parameters. U-Net, as a classic semantic segmentation model, can be applied in this process. This model consists of an encoding layer and a decoding layer, where the encoding layer is used for feature extraction, and the decoding layer outputs pixel-level classification prediction results.

[0046] S600: Extract the spatiotemporal features of all objects from the image within each hole and calculate the average value; Furthermore, the spatiotemporal features include morphological features, texture features, and daytime features.

[0047] The dataset consisting of normally focused objects is input into the semantic segmentation model for image segmentation. After accurately obtaining the object contours, all object morphological features, texture features, and daytime features are extracted from each hole image, and the average value is calculated.

[0048] Morphological features with application potential extracted from the segmentation results include area, perimeter, roundness, elongation, eccentricity, central moment, and concavity / convexity. For example, the formula for calculating roundness is: Where P represents the perimeter of the culture, A represents the area of ​​the culture, and R is the equivalent radius of the object.

[0049] The formula for calculating elongation is: in, Indicates the minimum diameter of the culture. Indicates the maximum diameter of the culture.

[0050] Texture features with application potential extracted from the segmentation results include gray-level co-occurrence matrix, gray-level run-length matrix, and gray-level region size matrix.

[0051] After adding drugs to the culture plate, the organoids or tumor spheres in the culture plate were observed every 24 hours. The diurnal features with application potential extracted from the segmentation results included the ratio of all the above features between every two observation days.

[0052] S700: Using the information in each well as a single sample, the average value of multiple features in the well as multiple features, and the real ATP cell viability as the label value, a regression machine learning network model is trained. The trained model is used to predict the viability of bright-field images.

[0053] Using the information within each well as a single sample, the average of multiple features within the well as the sample feature, and the true ATP activity as the label value, a regression machine learning network model is trained. The trained model can be used to predict the activity of bright-field images. The regression model dataset is ≥1000.

[0054] The principle of organoid or tumor spheroid viability prediction based on machine learning is as follows: Figure 12As shown, the precise location of organoids or tumor spheres in each well can be obtained through object detection and semantic segmentation. Therefore, spatiotemporal features in each well can be extracted and statistically analyzed. Each single-well image is used as a single sample, and a large number of spatiotemporal features are used as sample features. The ATP value corresponding to each well is used as a label. Feature filtering and PCA are used to remove redundancy and reduce dimensionality, reducing unnecessary computation. Regression machine learning network models, such as RF, XGBOOST, and CATBOOST, are trained. The trained model can be used to predict the vitality of bright-field images.

[0055] Due to the three-dimensional growth characteristics of organoids and tumor spheres, both normal and out-of-focus conditions can occur during the imaging process of organoids and tumor spheres, and are particularly noticeable in organoid images. Figure 5 To compare the bright-field morphology of single-hole images of normal and inactivated organoids cropped from large field-of-view images, the organoids in the completely inactivated images with an ATP luminescence value of 0 exhibited obvious characteristics such as small size and loss of organoid structure. Figure 6 Images of normal and completely inactivated tumor spheres are presented. Desquamated cells or small cell clusters are present around the tumor spheres, and different individual tumor spheres exhibit varying densities. Furthermore, the size and morphology of organoids and tumor spheres show high heterogeneity. Therefore, simple feature analysis based solely on quantity or area cannot accurately reflect the activity of organoids or tumor spheres. After segmenting individual organoids or tumor spheres, spatiotemporal features are extracted (hundreds of known morphological, textural, and diurnal features). To reduce the impact of redundant features in high-dimensional features on model generalization and to minimize unnecessary computation, high-dimensional features are screened and subjected to PCA dimensionality reduction.

[0056] In the feature selection and PCA dimensionality reduction feature engineering process, considering the limited sample size and potential feature redundancy, Spearman correlation analysis was performed between each feature and the ATP test results. Features showing a strong correlation with organoid or tumor sphere viability (e.g., Spearman correlation coefficient > 0.85) were selected for further analysis. Next, the selected features were standardized using the Z-score method, and Pearson correlation coefficients were calculated for the standardized features. Features with high correlations were then extracted for PCA dimensionality reduction. The extracted highly relevant feature matrix A is an array with m rows and l columns, where m is the total number of culture wells used in the 96-well plate (number of samples), and l is the number of features. It is the value of the j-th feature in the i-th culture well; The weights of the culture characteristic parameters are calculated through eigenvalue decomposition, and the culture characteristic parameters are normalized according to their weight coefficients. First, the covariance among the culture characteristic parameters is calculated to construct a covariance matrix B with l rows and l columns: In matrix B, cov() represents the covariance. Represents the j-th feature parameter. ; The eigenvalues ​​and eigenvectors of the covariance matrix are calculated through eigenvalue decomposition, forming a sequence of eigenvalues ​​from largest to smallest. The eigenvectors corresponding to the sorted eigenvalues ​​will be used as weighting coefficients for organ-like feature parameters.

[0057] The principal components corresponding to the eigenvalues ​​are retained from largest to smallest until the sum of the eigenvalues ​​of all principal components is reached. Greater than the threshold (such as setting) (The variance is 0.85). Combining these principal components with the original relevant features to form a new feature series completes the feature extraction process; where the variance ratio of each principal component is... for: in, is the i-th feature value, and l is the number of dimensions of the feature; Cumulative variance explained proportion: Among them, the proportion of cumulative variance explained is selected to reach a certain threshold. k principal components at time.

[0058] This invention proposes a novel method for high-throughput evaluation of the activity of three-dimensional organ models under bright-field conditions, based on a laboratory off-axis three-mirror large-field biological imaging system and combined with machine learning technology. The aim is to improve the efficiency of drug screening based on three-dimensional organ models and to provide the possibility for the continuous conduct of subsequent experiments.

[0059] Example 2 After implementing the complete process of the method provided in Example 1, the parameters for object detection, image segmentation, and the regression model have been determined. After acquiring images, researchers can input them into the software and ask questions. The large language model agent will perform inference based on natural language instructions and call various functional modules as needed, ultimately generating the corresponding response. The specific scheme is as follows: A high-throughput drug screening method based on large field-of-view imaging and machine learning is provided, applying the method provided in Example 1. For a detailed description of the method, please refer to the corresponding description in Example 1 above; it will not be repeated here. Figure 4 , Figure 13As shown, the method includes the following steps: S800: Acquires user-inputted images of the wells in the drug sensitivity test plate and natural language commands; The S900 large language model performs inference based on natural language instructions and calls various functional modules as needed to generate the corresponding response.

[0060] Image analysis process based on large language models, such as Figure 14 As shown. After the drug screening process is executed, it supports high-throughput drug screening experiments for clinical or laboratory personnel, who are typically physicians or researchers in biomedical fields. To facilitate easy operation and rapid acquisition of detailed drug screening results for personnel without programming skills, this method introduces a human-computer interaction technology based on a large language model. With the assistance of this technology, researchers can send natural language commands to the large language model agent, which will then perform inference and call the corresponding modules to generate the required results. Common user commands are as follows: Please count the number of normally focused objects in this series of images; Please calculate the average area and perimeter of the culture in each image, and plot the change curve from the low-concentration group to the high-concentration group based on these two features; Please calculate the culture activity value in each image, and plot the change curve from the low-concentration group to the high-concentration group; Please calculate the culture activity inhibition rate relative to the control group in each image, and plot the change curve from the low-concentration group to the high-concentration group; Please calculate the IC50 concentration value of the drug, etc.

[0061] This invention provides a high-throughput drug screening method based on a large field-of-view imaging system, combined with machine learning technology and three-dimensional organ models. This method can quantify the activity of three-dimensional organ models under bright field conditions, providing a higher-throughput experimental means for research in drug discovery, precision medicine and other fields, and can greatly accelerate the relevant research process.

[0062] Example 3 A high-throughput drug screening system based on large field-of-view imaging and machine learning applies the above-described method. For a detailed description of the method, please refer to the corresponding descriptions in the above-described method embodiments; they will not be repeated here. Figure 15 As shown, the system 110 includes a large field-of-view imaging system 111, a data processing module 112, and a human-computer interaction module 113 based on a large language model; wherein, like Figures 2-3 As shown, the optical lens of the large field-of-view imaging system includes a primary mirror, a secondary mirror, a third mirror, and a folding mirror. The primary mirror and the third mirror are freeform surfaces, the secondary mirror is a quadratic surface, and the folding mirror is a plane mirror. Light from the object side enters the optical lens and reaches the primary mirror. After being reflected by the primary mirror, it forms a first reflected light. The secondary mirror is placed in the reflected light path of the primary mirror and reflects the first reflected light from the primary mirror to form a second reflected light. The third mirror is placed in the reflected light path of the secondary mirror and reflects the second reflected light from the secondary mirror to form a third reflected light. The folding mirror is placed in the reflected light path of the third mirror and reflects the third reflected light from the third mirror to form a fourth reflected light. The fourth reflected light forms an image at the image plane. The data processing module is used to acquire images of each well of an antimicrobial susceptibility plate collected under a large field-of-view imaging system, and construct a target detection dataset. The antimicrobial susceptibility plate contains drug-treated tumor spheres or organoids. The module acquires the actual ATP cell viability in each well. It labels each image with focused and unfocused objects, and crops out single focused objects to construct a semantic segmentation dataset. The images are input into a target detection model for training, enabling the model to locate and classify focused and unfocused objects. The semantic segmentation dataset is input into the semantic segmentation model for image segmentation to accurately obtain object contours. All spatiotemporal features of objects are extracted from the images within each well, and their average values ​​are calculated. Using the information within each well as a single sample, the average value of multiple features within the well as multiple features, and the actual ATP cell viability as a label value, a regression machine learning network model is trained. The trained model is used to predict the viability of bright-field images. The human-computer interaction module based on the large language model is used to acquire the image inside the well of the drug sensitivity plate and natural language commands input by the user, perform inference based on the natural language commands, call various functional modules as needed, and finally generate the corresponding response.

[0063] Based on the technical solutions of the above embodiments, optionally, the step of acquiring images of each well of the drug sensitivity plate collected under a large field-of-view imaging system and constructing a target detection dataset further includes: When constructing the organoid dataset, a Z-axis pore-by-pore scan was performed, and multiple images were acquired for each pore.

[0064] Based on the technical solution of the above embodiments, optionally, the step of marking in-focus objects and out-of-focus objects on each image, and cropping out the in-focus single objects for constructing a semantic segmentation dataset includes: For each hole, perform focus-normal object detection on all images, and apply IOU matching and filtering strategies to images of different planes within the same hole: If the Intersection over Union (IOU) value of the same organoid is greater than a threshold in all images, it is considered a duplicate object. For the same object's detection bounding box region, only the one with the highest sharpness evaluation function is retained. All non-duplicate organoids are cropped for subsequent segmentation and regression processes. The IOU calculation formula is as follows: .

[0065] Based on the technical solutions of the above embodiments, optionally, the spatiotemporal features include morphological features, texture features, and daytime features.

[0066] Based on the technical solutions of the above embodiments, optionally, the morphological features include area, perimeter, roundness, elongation, eccentricity, central moment, and concavity / convexity; The texture features include a gray-level co-occurrence matrix, a gray-level run-length matrix, and a gray-level region size matrix; The daytime feature is the ratio of the morphological feature and the texture feature between every two observation times.

[0067] Based on the technical solution of the above embodiments, optionally, the method further includes steps of deduplication and dimensionality reduction of features through feature selection and PCA: Spearman correlation analysis was performed between each feature and ATP test results, and features showing a strong correlation with organoid or tumor sphere viability were selected for further analysis. The selected features are standardized using the Z-score method. The standardized features are then subjected to Pearson correlation coefficient calculation. Features with high correlation are extracted and subjected to PCA dimensionality reduction. The extracted highly correlated feature matrix A is an array with m rows and l columns, where m is the total number of culture wells used in the multi-well antimicrobial susceptibility plate, and l is the number of features. It is the value of the j-th feature in the i-th culture well; The weights of the culture characteristic parameters are calculated through eigenvalue decomposition, and the culture characteristic parameters are normalized according to their weight coefficients. The covariance between the culture characteristic parameters is calculated to construct a covariance matrix B with l rows and l columns. In matrix B, cov() represents the covariance. Represents the j-th feature parameter. ; The eigenvalues ​​and eigenvectors of the covariance matrix are calculated by eigenvalue decomposition, forming a sequence of eigenvalues ​​from maximum to minimum. The eigenvectors corresponding to the sorted eigenvalues ​​are used as weighting coefficients for organ-like feature parameters. The principal components corresponding to the eigenvalues ​​are retained from largest to smallest until the sum of the eigenvalues ​​of all principal components is reached. Greater than the threshold The feature extraction process is completed by combining these principal components with the original relevant features to form a new feature series; where the variance proportion of each principal component is... for: in, is the i-th feature value, and l is the number of dimensions of the feature; Cumulative variance explained proportion: Among them, the proportion of cumulative variance explained reaches a threshold. k principal components at time.

[0068] Example 4 A computer device 120, such as Figure 16 As shown, the system includes a memory 121, a processor 122, and a computer program 123 stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a high-throughput drug screening method based on large field-of-view imaging and machine learning. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0069] Example 5 A computer-readable storage medium, such as Figure 17 As shown, a computer program is stored thereon. When executed by a processor, the computer program implements the steps of a high-throughput drug screening method based on large field-of-view imaging and machine learning. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0070] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0071] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0072] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0073] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0074] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately by function as various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0075] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0081] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0082] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A high-throughput drug screening method based on large field-of-view imaging and machine learning, characterized in that, Includes the following steps: Images of each well of a drug-sensitive plate acquired using a large field-of-view imaging system are used to construct a target detection dataset; wherein, the drug-sensitive plate contains drug-treated tumor spheres or organoids. Obtain the actual ATP cell viability in each well of the test; Label each image with in-focus and out-of-focus objects, and crop out the in-focus single objects to construct a semantic segmentation dataset; The image is input into the object detection model for training, so that the model can locate and classify the in-focus target objects and the out-of-focus objects. The semantic segmentation dataset is input into the semantic segmentation model for image segmentation to accurately obtain object contours; Extract the spatiotemporal features of all objects from the image within each hole and calculate the average value; Using the information within each well as a single sample, the average of multiple features within the well as multiple features, and the actual ATP cell viability as the label value, a regression machine learning network model is trained. The trained model is then used to predict the viability of bright-field images.

2. The high-throughput drug screening method based on large field-of-view imaging and machine learning as described in claim 1, characterized in that, The step of acquiring images of each well of the drug-sensitive plate collected under a large field-of-view imaging system and constructing a target detection dataset further includes: When constructing the organoid dataset, a Z-axis pore-by-pore scan was performed, and multiple images were acquired for each pore.

3. The high-throughput drug screening method based on large field-of-view imaging and machine learning as described in claim 2, characterized in that, The steps of labeling in-focus and out-of-focus objects on each image, and cropping out individual in-focus objects for constructing a semantic segmentation dataset include: For each hole, perform focus-normal object detection on all images, and apply IOU matching and filtering strategies to images of different planes within the same hole: If the Intersection over Union (IOU) value of the same organoid is greater than a threshold in all images, it is considered a duplicate object. For the same object's detection bounding box region, only the one with the highest sharpness evaluation function is retained. All non-duplicate organoids are cropped for subsequent segmentation and regression processes. The IOU calculation formula is as follows: 。 4. The high-throughput drug screening method based on large field-of-view imaging and machine learning as described in claim 1, characterized in that: The spatiotemporal features include morphological features, textural features, and daytime features.

5. The high-throughput drug screening method based on large field-of-view imaging and machine learning as described in claim 4, characterized in that: The morphological features include area, perimeter, roundness, elongation, eccentricity, central moment, and concavity / convexity; The texture features include a gray-level co-occurrence matrix, a gray-level run-length matrix, and a gray-level region size matrix; The daytime feature is the ratio of the morphological feature and the texture feature between every two observation times.

6. The high-throughput drug screening method based on large field-of-view imaging and machine learning as described in claim 4, characterized in that, It also includes steps for feature selection and PCA to remove redundancy and reduce dimensionality of features: Spearman correlation analysis was performed between each feature and ATP test results, and features showing a strong correlation with organoid or tumor sphere viability were selected for further analysis. The selected features are standardized using the Z-score method. The standardized features are then subjected to Pearson correlation coefficient calculation. Features with high correlation are extracted and subjected to PCA dimensionality reduction. The extracted highly correlated feature matrix A is an array with m rows and l columns, where m is the total number of culture wells used in the multi-well antimicrobial susceptibility plate, and l is the number of features. It is the value of the j-th feature in the i-th culture well; The weights of the culture characteristic parameters are calculated through eigenvalue decomposition, and the culture characteristic parameters are normalized according to their weight coefficients. The covariance between the culture characteristic parameters is calculated to construct a covariance matrix B with l rows and l columns. In matrix B, cov() represents the covariance. Represents the j-th feature parameter. ; The eigenvalues ​​and eigenvectors of the covariance matrix are calculated by eigenvalue decomposition, forming a sequence of eigenvalues ​​from maximum to minimum. The eigenvectors corresponding to the sorted eigenvalues ​​are used as weighting coefficients for organ-like feature parameters. The principal components corresponding to the eigenvalues ​​are retained from largest to smallest until the sum of the eigenvalues ​​of all principal components is reached. Greater than the threshold The feature extraction process is completed by combining these principal components with the original relevant features to form a new feature series; where the variance proportion of each principal component is... for: in, It is the i-th eigenvalue. It is the number of dimensions of the feature; Cumulative variance explained proportion: Among them, the proportion of cumulative variance explained reaches a threshold. k principal components at time.

7. A high-throughput drug screening method based on large field-of-view imaging and machine learning, employing the method as described in any one of claims 1 to 6, characterized in that, Includes the following steps: Acquire images of the inside of the drug sensitivity test plate wells and natural language commands input by the user; The large language model performs inference based on natural language instructions and calls various functional modules as needed to generate the corresponding response.

8. A high-throughput drug screening system based on large field-of-view imaging and machine learning, using the method as described in any one of claims 1 to 7, characterized in that: It includes a large field-of-view imaging system, a data processing module, and a human-computer interaction module based on a large language model; among which, The optical lens of the large field-of-view imaging system includes a primary mirror, a secondary mirror, a third mirror, and a folding mirror. The primary mirror and the third mirror are freeform surfaces, the secondary mirror is a quadratic surface, and the folding mirror is a plane mirror. Light from the object side enters the optical lens and reaches the primary mirror. After being reflected by the primary mirror, it forms a first reflected light. The secondary mirror is placed in the reflected light path of the primary mirror and reflects the first reflected light from the primary mirror to form a second reflected light. The third mirror is placed in the reflected light path of the secondary mirror and reflects the second reflected light from the secondary mirror to form a third reflected light. The folding mirror is placed in the reflected light path of the third mirror and reflects the third reflected light from the third mirror to form a fourth reflected light. The fourth reflected light forms an image at the image plane. The data processing module is used to acquire images of each well of an antimicrobial susceptibility plate collected under a large field-of-view imaging system, and construct a target detection dataset. The antimicrobial susceptibility plate contains drug-treated tumor spheres or organoids. The module acquires the actual ATP cell viability in each well. It labels each image with focused and unfocused objects, and crops out single focused objects to construct a semantic segmentation dataset. The images are input into a target detection model for training, enabling the model to locate and classify focused and unfocused objects. The semantic segmentation dataset is input into the semantic segmentation model for image segmentation to accurately obtain object contours. All spatiotemporal features of objects are extracted from the images within each well, and their average values ​​are calculated. Using the information within each well as a single sample, the average value of multiple features within the well as multiple features, and the actual ATP cell viability as a label value, a regression machine learning network model is trained. The trained model is used to predict the viability of bright-field images. The human-computer interaction module based on the large language model is used to acquire the image inside the well of the drug sensitivity plate and natural language commands input by the user, perform inference based on the natural language commands, call various functional modules as needed, and finally generate the corresponding response.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images