Organ-like culture solution drug concentration control method based on deep reinforcement learning

Through a hybrid architecture model based on deep reinforcement learning, combined with convolutional neural network and long-term memory network, the drug concentration in organoid culture medium is dynamically adjusted, and the problem of insufficient drug concentration control accuracy and multi-objective control needs in the existing technology is solved. , achieving more efficient drug concentration control and optimization of organoid culture.

CN119040522APending Publication Date: 2024-11-29MEI HOSPITAL UNIV OF CHINESE ACAD OF SCI
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411270359.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing organoid culture medium drug concentration control method is insufficient in the face of complex and changing environments, and lacks dynamic adaptability, low data utilization rate, and cannot meet the multi-target control needs, which affects the accuracy and reliability of drug testing. .

Method used

Using a deep reinforcement learning method, a hybrid architecture model is constructed, combining convolutional neural network, long-term memory network and deep Q network, and dynamically adjust drug concentration using real-time sensor data, ensuring that drug concentration is at its highest through the growth rate and metabolic level objective function. Within the optimal range, precise control of multi-dimensional environmental parameters is achieved.

Benefits of technology

It significantly improves the accuracy and stability of drug concentration control, enhances the model's adaptability to complex situations, improves the success rate of organoid culture and the repeatability of experimental results, and provides a more reliable experimental environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119040522A_ABST
    Figure CN119040522A_ABST
Patent Text Reader

Abstract

The invention provides a deep reinforcement learning-based organoid culture solution drug concentration control method, which comprises the following steps of: constructing a deep reinforcement learning-based organoid culture solution drug concentration control model, and taking the current culture solution drug concentration, organoid physiological response data and environmental parameters as model input; outputting a set value of the flow speed adjustment amount of the drug delivery pump by taking the optimal drug concentration as a control target; a drug concentration control model is initially trained, then the trained model is integrated to a control unit of the system, sensor data is received and processed in real time, and through real-time feedback, deviation between actual drug concentration and target concentration is continuously compared, and organoid physiological response data is monitored, so that a control strategy is dynamically adjusted. According to the method, various factors influencing the drug concentration are comprehensively considered, and accurate control over the drug concentration of the organoid culture solution is achieved through the mixed architecture model of deep reinforcement learning. The model can process and analyze multi-dimensional complex data at the same time, so that a more accurate concentration adjustment decision is made.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of drug concentration control in organoid culture fluid, and in particular, relates to a method for controlling drug concentration in organoid culture fluid based on deep reinforcement learning. Background Art

[0002] In recent years, organoids have received widespread attention as an advanced technology that can simulate human organ functions and disease models in vitro. In the field of drug development and disease research, organoids are widely used in drug testing and toxicity assessment. The drug concentration in the organoid culture medium is crucial to the growth and functional expression of organoids. Accurately controlling the drug concentration in the culture medium can improve the accuracy and reliability of drug testing. However, existing drug concentration control methods have many shortcomings when dealing with the complex and changeable organoid culture environment, and it is difficult to meet the needs of high-precision drug delivery. The existing technology has the following defects: 1. Insufficient precision in drug concentration control: Traditional drug concentration control methods mainly rely on preset fixed parameters. It is difficult to adjust drug concentration in real time to cope with changes in environmental parameters during organoid culture, resulting in insufficient precision in drug concentration control, which affects the experimental effects of organoids.

[0003] 2. Lack of dynamic adaptability: Existing methods usually use simple feedback control mechanisms, which cannot effectively capture the dynamic changes in the growth process of organoids, especially when faced with different responses of different organoids to drug concentrations, and cannot flexibly adjust drug concentrations.

[0004] 3. Low data utilization: Although a large amount of environmental parameters and experimental data can be collected during the organoid culture process, existing technologies fail to fully utilize these data for precise drug concentration regulation, resulting in low data utilization and inability to achieve optimal drug delivery strategies.

[0005] 4. Insufficient support for multi-target control needs: In the process of organoid culture, in addition to drug concentration, the synergistic effects of multiple environmental parameters need to be considered. However, when facing multi-target control needs, existing technologies lack effective processing mechanisms and cannot meet the control needs of different targets at the same time. Summary of the invention

[0006] In view of the above problems, the present invention provides a method for controlling drug concentration in organoid culture medium based on deep reinforcement learning, comprising the following process: Construct a drug concentration control model for organoid culture medium based on deep reinforcement learning, using the current drug concentration of the culture medium, organoid physiological response data, and environmental parameters as model inputs, the optimal drug concentration as the control target, and outputting the set value of the flow rate adjustment of the drug delivery pump; Initially train the drug concentration control model, then integrate the trained model into the system's control unit to receive and process sensor data in real time. Through real-time feedback, continuously compare the deviation between the actual drug concentration and the target concentration, and monitor the physiological response data of the organoids to dynamically adjust the control strategy. The physiological response data of organoids include growth rate and metabolic level; the growth rate objective function is defined, and the growth rate is controlled by controlling the drug concentration. Reach the maximum value while ensuring that the drug concentration is within the allowed range; set the metabolic level constraint function to ensure that the metabolic level of the organoid does not deviate from the baseline level by more than the set deviation , to ensure that the effect of drug concentration on metabolic levels is within an acceptable range.

[0007] Preferably, the original data set collection process for initial training of the drug concentration control model is: Drug concentration data for organoid culture medium : Use liquid chromatography-mass spectrometry HPLC-MS to measure the drug concentration in the culture medium every few minutes and record the concentration at different time points Drug concentration , construct a time series dataset; Growth rate : Real-time cell analyzer RTCA was used to monitor the size changes of organoids and calculate the size of each time interval Relative growth rate:

[0008] Metabolic level :Use fluorescence-based glucose and lactate metabolism sensors to monitor in real time, record glucose consumption rate and lactate production rate, and comprehensively calculate the metabolic level index; Multi-dimensional environmental parameters: including temperature, pH value, oxygen concentration, and record the corresponding relationship between these parameters and drug concentration: { , , } →

[0009] All the collected original data sets are recorded as .

[0010] Preferably, the obtained original data set needs to be preprocessed, including data cleaning, data standardization and label determination, and divided into training set, validation set and test set; The data cleaning includes outlier detection and processing using the Z-score method, noise removal using the Savitzky-Golay filter, and missing value filling; The data standardization is a normalization process; The label adjudication includes the definition of target drug concentration labels for the culture medium, the setting of organoid physiological response targets, and label calibration.

[0011] Preferably, the culture medium target drug concentration label is specifically defined as: Determine the effect of drugs in organoid culture to set the optimal drug concentration range for each combination , the optimal concentration range represents the range in which the drug concentration can achieve the best effect in organoid culture, and the objective function is constructed as follows:

[0012] in, is the actual drug concentration after normalization, is the optimal drug concentration, and are the maximum and minimum values ​​of the optimal concentration range respectively; this objective function measures the actual drug concentration With the optimal concentration The closer the value of the objective function is to 1, the closer the actual concentration is to the optimal concentration.

[0013] Preferably, the organoid physiological response target formulation includes defining a growth rate objective function and setting a metabolic level constraint function; The growth rate objective function is defined as:

[0014] in, For time The growth rate target at the moment, For in time The actual growth rate at that moment, For in time The drug concentration at the time, and are the minimum and maximum allowable drug concentrations respectively; the goal is to control the drug concentration to increase the growth rate to achieve the maximum value while ensuring that the drug concentration is within the permissible range; The metabolic level constraint function is set as follows:

[0015] in, For time The metabolic level constraints at all times, For time Baseline metabolic level at the time, is the allowable deviation range.

[0016] Preferably, the organoid culture fluid drug concentration control model is a designed hybrid architecture, including an input module, a convolutional neural network module, a long short-term memory network module and an output module; The input module receives multi-dimensional data, including current drug concentration , Growth rate , metabolic level And environmental parameters, including temperature , pH value and oxygen concentration ; These input data are organized into a multidimensional tensor , expressed as:

[0017] The convolutional neural network module is used for feature extraction, for extracting spatial features in input data, especially the complex relationship between environmental parameters and drug concentration; The long short-term memory network module is used for feature enhancement, effectively capturing and retaining long-term and short-term dependencies in time series data; The output module uses a reinforcement learning method to output the next action to be taken, that is, the set value of the flow rate adjustment amount of the drug delivery pump, according to the current state environment, including the current drug concentration, organoid growth rate, metabolic level and environmental parameters.

[0018] Preferably, the convolutional neural network module comprises a plurality of convolutional layers and pooling layers; Tensor data , as non-sequential data for convolution operation:

[0019] in, It is The characteristics of the layer, indicating the position in the layer The characteristic value of It is The convolution kernel of the layer is used to extract specific types of features. Represents the relative position of the elements in the convolution kernel, is the output of the previous layer, that is, the input data or the features of the previous layer, is a bias term used to adjust the convolution result. It is an activation function used to introduce nonlinear features and accelerate convergence; Through convolution operations, the convolutional neural network module can extract local features in the input data and further reduce the feature dimension and computational complexity through the pooling layer.

[0020] Preferably, the long short-term memory network module includes a forget gate, an input gate and an output gate; Forget Gate Determine the past information degree of retention; Input Gate and candidate cell states Determines the current input The importance of, and how to update cell state; Cell state The choice of forget gate and input gate are combined to form the main memory of the current time step; Output Gate Combined with the cell state, it determines the hidden state output for the next time step What is; The LSTM network module effectively captures and retains long-term and short-term dependencies in time series data through these gating mechanisms.

[0021] Preferably, the output module uses a deep Q network DQN, and uses a reinforcement learning method to determine the next action to be taken, that is, the set value adjustment amount of the flow rate adjustment amount of the drug delivery pump, according to the current state environment, that is, the current drug concentration, organoid growth rate, metabolic level and environmental parameters, so as to achieve precise control of the drug concentration; DQN continuously iterates and updates the Q value function, and finally finds a strategy that maximizes the cumulative reward, wherein the Q value function:

[0022] in, Indicates the current time status, i.e., current drug concentration, organoid growth rate, metabolic levels, and environmental parameters; Indicates the current state The action taken is the set value adjustment amount of the drug delivery pump flow rate adjustment amount; Indicates the current time The reward obtained, i.e., the immediate response of the organoids to the drug concentration; is the discount factor, which is used to measure the importance of future returns. is the total number of iterative updates; Use the epsilon-greedy iteration strategy to The probability of choosing the action with the largest Q value is:

[0023] The largest action selected Converting into specific control instructions takes into account the current state and target state of drug concentration, as well as the physical limitations of the system; The output module finally generates a normalized control instruction ,in: A negative value indicates a decrease in drug concentration; a positive value indicates an increase in drug concentration; the larger the absolute value, the greater the adjustment; Control instructions Through a mapping function Converted to actual drug delivery pump flow rate adjustment :

[0024] in, is the maximum drug delivery pump flow rate allowed by the system, The function ensures that the output is within a reasonable range; Finally, the controller will It is sent as a control target instruction to achieve precise regulation of the drug concentration in the culture medium.

[0025] Preferably, in the process of learning the drug concentration control model of the organoid culture medium, a combination of simulation experiments and actual feedback is used; Build a simulation environment that can simulate the growth of different types of organoids under various conditions and the effects of changing drug concentrations on organoids; the model learns the optimal drug concentration control strategy by interacting with the environment; In actual organoid culture experiments, data are collected regularly, including drug concentrations, organoid growth conditions, and metabolic levels. These data are used to evaluate the performance of the model in the actual environment and serve as a basis for model updates; Use the collected actual experimental data to fine-tune the model online. Use the incremental learning method to update the model parameters using small batches of actual data. During the fine-tuning process, keep the knowledge learned by the model in the simulation environment while adapting to the characteristics of the actual environment. At the same time, domain adaptation technology is adopted and adversarial training methods are used to enable the model to learn to extract common features from simulated data and actual data, thereby improving the generalization ability of the model in actual environments.

[0026] Compared with the prior art, the present invention has the following beneficial effects: (1) A hybrid architecture model based on deep reinforcement learning is proposed, which combines the advantages of convolutional neural networks (CNN), long short-term memory networks (LSTM) and deep Q networks (DQN) for precise control of drug concentration in organoid culture medium. The model can simultaneously process multi-dimensional input data, including current drug concentration, organoid physiological response data (growth rate and metabolic level) and environmental parameters (temperature, pH value and oxygen concentration), thereby comprehensively considering various factors affecting drug concentration.

[0027] Through the hybrid architecture model of deep reinforcement learning, precise control of drug concentration in organoid culture medium is achieved. The model can simultaneously process and analyze complex data in multiple dimensions, including drug concentration, physiological response, and environmental parameters, to make more accurate concentration adjustment decisions. This comprehensive data processing capability significantly improves the accuracy and stability of drug concentration control, providing a more reliable experimental environment for organoid research.

[0028] (2) Organoid physiological response target formulation and objective function are used to formulate precise control of drug concentration labels and organoid response targets, ensuring that drug concentrations can be adjusted within the optimal range while controlling the growth rate and metabolic level of organoids within the expected range. These steps are key parts to ensure the accuracy of experimental data and optimization of experimental effects. At the same time, the label calibration method based on Bayesian optimization significantly improves the accuracy and reliability of data labels. By combining expert knowledge, experimental verification and mathematical optimization, this method can dynamically adjust labels to make them closer to actual conditions. This method not only improves the effect of model training, but also enhances the model's ability to adapt to various complex situations during organoid culture, so that in practical applications, drug concentrations can be controlled more accurately, the success rate of organoid culture and the repeatability of experimental results can be improved.

[0029] (3) By combining adaptive reinforcement learning, imitation learning and actual feedback, an innovative strategy for dynamically optimizing the drug concentration control of organoid culture medium is proposed. Adaptive reinforcement learning uses dynamically adjusted learning rates and exploration strategies to enhance the generalization ability and robustness of the model; imitation learning combines expert knowledge and reinforcement learning to accelerate the training process and improve initial performance; the combination of simulation environment and actual experimental data ensures the optimization and adaptation of the model in the real environment. This method comprehensively considers the complexity of training and practical application, and achieves precise drug concentration control and optimization of organoid culture. Ultimately, this comprehensive approach not only improves the generalization ability and practical application effect of the model, but also ensures the success rate of the organoid culture process and the repeatability of experimental results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an overall principle block diagram of the method for controlling drug concentration in organoid culture medium of the present invention.

[0031] Figure 2 Schematic diagram of the interaction process between the Q network and the environment in deep reinforcement learning of the present invention.

[0032] Figure 3 Schematic diagram of the iterative trajectory of the state space and action space in the deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The present invention arranges high-precision drug concentration sensors, organoid physiological state monitoring sensors and environmental parameter sensors in the organoid culture system of the microfluidic chip to monitor the drug concentration of the culture solution and the growth and metabolic state of the organoid in real time. At the same time, a real-time data acquisition system is constructed to ensure the cleaning and standardization of the data and to have an outlier detection function. On this basis, the trained deep reinforcement learning model is integrated into the central control unit of the system to ensure that the sensor data can be received and processed in real time. The control instructions output by the model are used to adjust the flow rate of the drug delivery pump to optimize the drug concentration control.

[0035] A closed-loop control system was implemented, in which the deep reinforcement learning controller model optimized the drug delivery pump flow rate by comprehensively analyzing the drug concentration, organoid growth rate, and metabolic level status, while setting constraints such as the metabolic level being within the allowable range and ensuring that the organoid growth rate reaches the maximum value. Through the real-time feedback mechanism, the system can continuously compare the deviation between the actual drug concentration and the target concentration, and monitor the growth and metabolic status of the organoids to dynamically adjust the control strategy, thereby achieving precise drug concentration control and optimization of the physiological state of the organoids. The overall process of the present invention is as follows: Figure 1 shown.

[0036] 1. Data Collection and Labeling This phase aims to collect, process, and label data for training the organoid culture fluid drug concentration control model. High-quality, diverse data is the basis for building an effective organoid culture fluid drug concentration control model, so this process is crucial to the performance of the model.

[0037] 1. Raw data collection: (1) Drug concentration data of organoid culture medium ( ): The drug concentration in the culture medium was measured every 30 minutes using a high performance liquid chromatography-mass spectrometer (HPLC-MS), and the concentrations of the drugs at different time points were recorded. Drug concentration , construct a time series dataset.

[0038] (2) Physiological response data of organoids: Growth rate ( ): Real-time cell analyzer (RTCA) was used to monitor the size changes of organoids and calculate the size of each time interval. Relative growth rate:

[0039] Metabolic level ( ): Fluorescence-based glucose and lactate metabolism sensors are used for real-time monitoring, and the glucose consumption rate and lactate production rate are recorded to comprehensively calculate the metabolic level index.

[0040] (3) Data on the relationship between multidimensional environmental parameters and drug concentrations: temperature( ): Use precision temperature sensor with accuracy of ±0.1°C; pH value ): Using micro pH electrode, accuracy ±0.01; Oxygen concentration ( ): Apply fluorescence quenching oxygen sensor with an accuracy of ±0.1%.

[0041] Record the corresponding relationship between these parameters and drug concentration: , , } →

[0042] All the collected original data sets are recorded as .

[0043] 2. Data preprocessing: (1) Data cleaning: Z-score method outlier detection and processing: Outliers are values ​​that deviate significantly from other data points in a data set. They may be caused by incorrect records or other factors. These outliers can be identified using the Z-score method. The Z-score represents the difference between an observation and the mean of the data set, calculated in units of standard deviation. The formula is:

[0044] in, is the observed value, is the mean of the data set, is the standard deviation of the data set.

[0045] When the absolute value of the Z-score When , the value is considered an outlier. For these outliers, the moving median (i.e. the median within a certain window) is used instead to reduce the impact of outliers on data analysis.

[0046] Savitzky-Golay filter noise removal: This is a filter commonly used to smooth time series data. It can effectively remove noise while maintaining the trend and volatility of the data. It is calculated by a weighted sum:

[0047] in, is the filter coefficient, is the original data point, is the filter window size.

[0048] Missing value filling: Short-term missing: Linear interpolation: When data is missing for a short period of time, linear interpolation is used to estimate the missing values. The formula is as follows:

[0049] in: and are two adjacent known data points, is the position of the point to be estimated, is an estimated value.

[0050] Long-term missing: Multiple imputation: When data are missing for a long time, simple interpolation methods may not be accurate enough. Multiple interpolation can be used to fill in the missing data by simulating multiple possible values ​​and performing subsequent analysis based on these simulated values.

[0051] (2) Data standardization Normalization of drug concentrations: In order to make the drug concentration data comparable on a uniform scale, a normalization method is applied:

[0052] in, is the original concentration value. is the minimum concentration value in the data set, is the maximum concentration value in the data set. The normalized concentration Will be between 0 and 1.

[0053] The same method was used to standardize the temperature, pH value and oxygen concentration in the environmental parameters. , , .

[0054] All the preprocessed data sets are recorded as .

[0055] 3. Label ruling: (1) Target drug concentration label definition: Based on the domain experts’ expertise in various organoid and drug combinations, combined with analysis of past experimental data to determine the effect of drugs in organoid culture, the optimal drug concentration range for each combination is set , the optimal concentration range represents the range in which the drug concentration can achieve the best effect in organoid culture. The objective function is constructed in this way:

[0056] in, is the actual drug concentration after normalization, is the optimal drug concentration, and are the maximum and minimum values ​​of the optimal concentration range, respectively.

[0057] This objective function measures the actual drug concentration With the optimal concentration The closer the value of the objective function is to 1, the closer the actual concentration is to the optimal concentration.

[0058] (2) Setting physiological response targets for organoids: Define the growth rate objective function:

[0059] in, For time The growth rate target at the moment, For in time The actual growth rate at that moment, For in time The drug concentration at the time, and are the minimum and maximum permissible drug concentrations, respectively.

[0060] The goal is to control the drug concentration to increase the growth rate to achieve the maximum value while ensuring that the drug concentration is within the permissible range.

[0061] Metabolic level constraint function:

[0062] in, For time The metabolic level constraints at all times, For time Baseline metabolic level at the time, is the allowable deviation range.

[0063] This constraint ensures that the metabolic levels of the organoids do not deviate from baseline levels by more than a set deviation. , to ensure that the effect of drug concentration on metabolic levels is within an acceptable range.

[0064] These definitions and objective functions are used to develop precise control of drug concentration labels and targets for organoid responses, ensuring that drug concentrations can be adjusted within the optimal range while controlling the growth rate and metabolic levels of organoids within the expected range. These steps are key parts of ensuring experimental data accuracy and optimizing experimental results.

[0065] (3) Label calibration: a. Invite domain experts to review initial labels: First, experts with relevant field knowledge are invited to review the initial labels of the dataset. Based on their professional knowledge, the experts will evaluate whether the labels are correct and whether they need to be adjusted or corrected. This can improve the accuracy of the labels and reduce annotation errors.

[0066] b. Conduct a small-scale validation experiment to compare the actual results with the predetermined labels: Conduct a small-scale experiment to verify whether the labels of the dataset can accurately reflect the actual situation. Compare the experimental results with the predetermined labels to further confirm the accuracy of the labels. If differences are found, it may be necessary to re-examine the rationality of the labels.

[0067] c. Use Bayesian optimization method to iteratively adjust the label: Bayesian optimization is a probability-based optimization method used to make optimal decisions under uncertain conditions. The specific steps are as follows: Posterior probability : Represents the probability of a label after observing the data, which is the result we hope to obtain through calibration.

[0068] Likelihood function : It indicates the probability of data occurrence given a label, which reflects the degree of match between the label and the data.

[0069] Prior probability : It represents the prior belief about a label before there is any observed data, that is, the initial hypothesis of the label.

[0070] The Bayesian formula is:

[0071] By continuously iteratively adjusting the labels, the posterior probability of the labels is maximized when the data is observed, thereby calibrating the labels. This systematic data collection, preprocessing, and labeling method can obtain high-quality, standardized data sets, laying a solid foundation for subsequent deep reinforcement learning model training. This method not only ensures the accuracy and consistency of the data, but also fully considers the complexity and diversity of organoid culture, providing rich learning materials for the model.

[0072] All the labels obtained after the judgment are recorded as .

[0073] 4. Dataset construction: In order to divide the labeled data into training set, validation set and test set, and prepare for model training, evaluation and testing, it is necessary to construct a dataset: the labeled dataset is recorded as , the training set, validation set, and test set are divided into 8:1:1 ratios.

[0074] In order to divide the labeled data into training set, validation set and test set to prepare for model training, evaluation and testing, the dataset needs to be According to the ratio of 8:1:1, the specific steps are as follows: The labeled dataset Divided in the following proportions: 80% of the data is used as training set 10% of the data is used as a validation set. 10% of the data is used as a test set (Test Set) Such a division can ensure that there is sufficient data for parameter adjustment and optimization during the model training process, and perform evaluation of the validation set and final performance verification of the test set after model training to ensure the generalization ability and effectiveness of the model.

[0075] 2. Establishment of a drug concentration control model for organoid culture medium based on deep reinforcement learning This model adopts an innovative hybrid architecture that combines the advantages of convolutional neural networks (CNN), long short-term memory networks (LSTM) and deep Q networks (DQN) to achieve precise control of drug concentration in organoid culture medium.

[0076] 1. Input module design: The input module receives multi-dimensional data, including: Current drug concentration ; Organoid physiological response data: growth rate and metabolic levels ; Environmental parameters: Temperature , pH value and oxygen concentration .

[0077] These input data are organized into a multidimensional tensor , expressed as:

[0078] 2. Convolutional Neural Network (CNN) Module: The CNN module is used to extract spatial features from the input data, especially the complex relationship between environmental parameters and drug concentration. This module contains multiple convolutional layers and pooling layers. Tensor data , as non-sequential data for convolution operation:

[0079] in, It is The characteristics of the layer, indicating the position in the layer The characteristic value of . It is The convolution kernel of the layer is used to extract specific types of features. Represents the relative position of the elements in the convolution kernel, is the output of the previous layer, that is, the input data or the features of the previous layer, is a bias term used to adjust the convolution result. It is an activation function used to introduce nonlinear features and accelerate convergence.

[0080] Through convolution operations, CNN can extract local features in the input data and further reduce the feature dimension and computational complexity through pooling layers.

[0081] 3. Long Short-Term Memory Network (LSTM) Module: The LSTM module is used to process and predict dynamic changes in time series data, especially the long-term response of organoids to drug concentrations.

[0082] Forget Gate:

[0083] in, is the current time step The forget gate output determines how much information from the previous time step needs to be retained to the current time step. is the Sigmoid activation function, which compresses the output to the range of [0,1]. is the weight matrix of the forget gate, which is used to process input and hidden states. is the current time step The input of the convolutional neural network module is Sample data for ) and the hidden state at the previous time step connection, is the bias term of the forget gate.

[0084] Input Gate:

[0085] in, is the current time step The input gate output controls how much new information will be added to the cell state. and are the weights and biases of the input gate.

[0086] Candidate cell states:

[0087] in, is the current time step The candidate cell state is the new information determined by the input gate. and are the weights and biases of the candidate cell states.

[0088] Updated cell status:

[0089] in, is the current time step The cell state, is the output of the forget gate, determining The degree of information retention in is the cell state at the previous time step, is the output of the input gate, which determines the state of the candidate cell The importance of.

[0090] Output Gate:

[0091] in, is the current time step The output gate output determines the current cell state The degree of influence on the hidden state, and are the weight and bias of the output gate.

[0092] Hidden state:

[0093] in, is the current time step The hidden state is output to the next time step or as the final output. is the current time step The cell state The value after the activation function.

[0094] Overall process: Forget Gate Determine the past information degree of retention.

[0095] Input Gate and candidate cell states Determines the current input The importance of, and how to update the cell state.

[0096] Cell state The choice of the forget gate and the choice of the input gate are combined to form the main memory of the current time step.

[0097] Output Gate Combined with the cell state, it determines the hidden state output for the next time step What is it.

[0098] LSTM uses these gating mechanisms to effectively capture and preserve long-term and short-term dependencies in time series data.

[0099] 4. Output module Use Deep Q Network (DQN): DQN uses reinforcement learning methods to decide the next action to be taken (the set value adjustment of the drug delivery pump flow rate adjustment) according to the current state environment (current drug concentration, organoid growth rate, metabolic level and environmental parameters), thereby achieving precise control of drug concentration and ultimately achieving the optimal physiological response of the organoid.

[0100] In reinforcement learning, the Q-value function is the core, used to evaluate the Next action The expected return. DQN continuously iterates and updates the Q value function, and finally finds the strategy that maximizes the cumulative return. Among them, the Q value function:

[0101] in, Indicates the current time Status of: current drug concentration, organoid growth rate, metabolic levels, and environmental parameters; Indicates the current state The following actions are taken: the set value adjustment amount of the drug delivery pump flow rate adjustment amount; Indicates the current time The rewards: Immediate responses of organoids to drug concentrations; is the discount factor, which is used to measure the importance of future returns. is the total number of iterative updates. The interaction process between the Q network and the environment in deep reinforcement learning is as follows: Figure 2 shown.

[0102] The purpose of the output module is to generate control instructions from the controller to the drug delivery module to achieve precise control of drug concentration. Specifically: Using the output of the LSTM module , the output contains the long-term dependency information of time series data.

[0103] Combined with Q value function The calculation results are as follows: is the current state, is a possible action. The iterative trajectory of the state space and action space in deep reinforcement learning is as follows Figure 3 shown.

[0104] Use the epsilon-greedy iterative strategy combined with the output of the LSTM module to balance exploration and exploitation: by The probability of choosing the action with the largest Q value is:

[0105] by An action is chosen randomly with probability , to ensure a certain degree of exploration.

[0106] The selected action This process takes into account the current state and target state of drug concentration, as well as the physical limitations of the system.

[0107] The output module finally generates a normalized control instruction ,in: A negative value indicates a decrease in drug concentration; a positive value indicates an increase in drug concentration; the larger the absolute value, the greater the adjustment.

[0108] Control instructions Through a mapping function Converted to actual drug delivery pump flow rate adjustment :

[0109] in, is the maximum drug delivery pump flow rate allowed by the system, The function ensures that the output is within a reasonable range.

[0110] Finally, the controller will It is sent as an instruction to the drug delivery module to achieve precise control of the drug concentration in the culture medium.

[0111] In this way, the output layer effectively transforms the decisions of the deep reinforcement learning model into actual control behaviors, making full use of the previous LSTM's processing of time series information and Q learning's selection of optimal actions, thereby achieving intelligent and precise control of the drug concentration in the organoid culture medium.

[0112] 3. Organoid culture medium drug concentration control model training 1. Adaptive reinforcement learning During model training, an adaptive reinforcement learning strategy is used to dynamically adjust the learning rate and exploration strategy to ensure that the model can adapt to different types of organoids and drug combinations. This approach can improve the generalization ability and robustness of the model.

[0113] (1) Dynamic learning rate adjustment: Use the Adam optimizer in combination with a learning rate decay strategy. The initial learning rate is set to 0.001 and then dynamically adjusted based on the performance on the validation set. Specifically, the cumulative reward on the validation set is monitored, and if there is no improvement for five consecutive epochs, the learning rate is reduced to 80% of the original. This approach can maintain a faster learning speed in the early stages of training, while making more refined parameter adjustments in the later stages.

[0114] (2) Adaptive exploration strategy: The ε-greedy strategy is used for exploration, but the ε value is dynamically adjusted according to the training progress and model performance. The initial ε value is set to 0.9, indicating a 90% probability of random exploration. As training progresses, the ε value gradually decreases, but the rate of decrease is dynamically adjusted according to the performance of the model on the validation set. If the model performance stagnates, the ε value will be temporarily increased to increase exploration to escape the local optimum.

[0115] (3) Prioritized Experience Replay: Prioritized Experience Replay is used to assign a priority to each experience sample based on the TD error. This ensures that the model can learn important samples with high TD errors more frequently, improving learning efficiency.

[0116] 2. Combining imitation learning with reinforcement learning To speed up the learning process and ensure that the model has good initial behavior, imitation learning is combined with reinforcement learning. This approach allows the model to first learn from expert knowledge and then further optimize its strategy through reinforcement learning.

[0117] (1) Behavior cloning: Using the dataset produced by S1, we first train a supervised learning model to imitate expert behavior. This model uses the same neural network architecture as the main reinforcement learning model, but is only used to predict the actions that the expert will take in a given state. We use the cross entropy loss function to train this model to ensure that it can accurately replicate expert behavior.

[0118] (2) Pre-training reinforcement learning model: The weights of the behavior cloning model are used to initialize the reinforcement learning model. This provides a good starting point for reinforcement learning, so that the model has reasonable behavior at the beginning of training.

[0119] (3) Hybrid learning strategy: A hybrid strategy is used during reinforcement learning. When making decisions, the model has a certain probability (initially set to 30%, which gradually decreases as the training progresses) of following the advice of the pre-trained model instead of relying entirely on the current Q-value estimate. This approach helps maintain stability while exploring new strategies.

[0120] (4) Progressive policy distillation: As reinforcement learning proceeds, the current policy is regularly compared with the initial expert policy. If the current policy is significantly better than the expert policy, the expert model is updated to achieve policy distillation. This ensures that the model can go beyond the initial expert knowledge while retaining valuable expert experience.

[0121] 3. Combining simulation experiments with actual feedback In order to ensure the effectiveness of the model in practical applications, the training in the simulation environment is combined with the feedback of actual experimental data to achieve continuous optimization and adaptation of the model.

[0122] (1) Simulation environment construction: Based on the dataset and model established in the previous article, a detailed simulation environment was constructed. This environment can simulate the growth of different types of organoids under various conditions, as well as the effects of drug concentration changes on organoids. The simulation environment uses the objective function and constraints defined in the previous article to evaluate the performance of the model.

[0123] (2) Simulation training: In a simulation environment, the model learns the optimal drug concentration control strategy by interacting with the environment. A deep Q-learning algorithm is used in combination with the adaptive reinforcement learning technique described above to perform a large number of iterative training in a simulation environment.

[0124] (3) Actual experimental data collection: In actual organoid culture experiments, data are collected regularly, including drug concentration, organoid growth status, and metabolic levels. These data are used to evaluate the performance of the model in the actual environment and serve as the basis for model updates.

[0125] (4) Online model fine-tuning: Use the collected actual experimental data to fine-tune the model online. This process uses an incremental learning method to update model parameters using small batches of actual data. During the fine-tuning process, special attention is paid to maintaining the knowledge learned by the model in the simulation environment while adapting to the characteristics of the actual environment.

[0126] (5) Domain adaptation technology: In order to address the possible differences between the simulation environment and the actual environment, domain adaptation technology is used. Specifically, adversarial training methods are used to enable the model to learn to extract common features from simulation data and actual data, thereby improving the model's generalization ability in the actual environment.

[0127] (6) Continuous learning and updating: A continuous learning loop is established to regularly update the model using newly collected experimental data. This process includes: a) evaluating the performance of the model on new data b) identifying cases where the model performs poorly c) using these cases for targeted training d) updating model parameters without destroying existing knowledge (7) Safety constraint integration: In practical applications, rule-based safety constraints are integrated. Even based on the model's recommendations, the system checks whether pre-defined safety thresholds (such as maximum allowable drug concentration) are violated. This provides an additional layer of safety assurance, ensuring that the model's decisions do not lead to potentially dangerous situations.

[0128] Through this comprehensive training method, the model can be continuously optimized in theory and practice to adapt to different types of organoids and drug combinations, while maintaining good generalization and practical application effects. This method not only uses a large amount of simulation data for extensive learning, but also ensures the effectiveness of the model in a real environment through feedback from actual experimental data, thus providing a powerful and reliable solution for the precise control of drug concentration in organoid culture fluid.

[0129] 4. Model Deployment and Application This stage aims to deploy the trained deep reinforcement learning model into the organoid culture system and achieve real-time control and optimization of drug concentration.

[0130] 1. Sensor layout and data acquisition system deployment: (1) In the microfluidic chip-based organoid culture system, the following sensors are placed at key locations in the drug circulation pipeline: a) High-precision drug concentration sensor: A miniaturized high-performance liquid chromatography-mass spectrometry (HPLC-MS) is used to monitor the drug concentration in the culture medium in real time.

[0131] b) Sensors for monitoring the physiological status of organoids: including real-time cell analyzers (RTCA) for monitoring changes in organoid size, and fluorescence-based glucose and lactate metabolism sensors.

[0132] c) Environmental parameter sensors: including precision temperature sensor (accuracy ±0.1°C), micro pH electrode (accuracy ±0.01) and fluorescence quenching oxygen sensor (accuracy ±0.1%).

[0133] (2) Build a real-time data acquisition system: Based on the data acquisition and preprocessing system designed in the previous article, it is possible to collect, clean and standardize the data from various sensors in real time. The system should have anti-interference capabilities and be able to perform preliminary outlier detection and processing.

[0134] 2. Controller model integration: (1) Integrate the trained deep reinforcement learning model into the central control unit of the organoid culture system. The control unit should have sufficient computing power to support real-time reasoning and decision-making of the model.

[0135] (2) Realize seamless integration of the model and the data acquisition system to ensure that the model can receive preprocessed sensor data in real time.

[0136] (3) Based on the model output interface designed above, the decision results of the model are converted into specific control instructions: the flow rate of the drug delivery pump is adjusted to the optimal value.

[0137] 3. Closed-loop control system implementation: (1) Based on the multi-objective control strategy established above with drug concentration and organoid physiological state as optimization objectives, it includes: growth rate objective function and metabolic level constraint function.

[0138] (2) Implementing a controller model based on deep reinforcement learning: a) State space: including current drug concentration, organoid growth rate, metabolic levels, and environmental parameters.

[0139] b) Action space: includes the set value adjustment of the drug delivery pump flow rate adjustment.

[0140] c) Reward function: Based on a multi-objective control strategy, it comprehensively considers the drug concentration control accuracy, organoid growth status and metabolic level.

[0141] (3) Implementing feedback mechanism: a) Compare the deviation between actual drug concentration and target concentration in real time.

[0142] b) Monitor organoid growth rate and metabolic profile to ensure they are within expected ranges.

[0143] c) Continuously optimize control strategies based on deviations and monitoring results.

[0144] 4. Safety mechanism and exception handling: (1) Implement multiple security protection mechanisms: a) Set safe upper and lower limits for drug concentration to prevent over- or under-dosing.

[0145] b) Monitor the physiological indicators of organoids, set warning thresholds, and detect abnormal situations in a timely manner.

[0146] c) Achieve a safe control range of environmental parameters to prevent damage to organoids due to extreme conditions.

[0147] (2) Design abnormal situation handling process: a) Implement an automatic alarm system to promptly notify experimenters when an abnormality is detected.

[0148] b) Develop emergency control strategies that can quickly restore the system to a safe state under abnormal circumstances.

[0149] c) Establish a logging system to record all operations and abnormal situations in detail to facilitate subsequent analysis and improvement.

[0150] 5. Model online learning and optimization: (1) Implement the online learning mechanism of the model: a) Continuously collect new data samples during the control process.

[0151] b) Regularly update model parameters using newly collected data to improve model adaptability and control accuracy.

[0152] (2) Design model performance evaluation indicators: a) Drug concentration control precision: evaluates the average deviation between the actual concentration and the target concentration.

[0153] b) Organoid growth status: Monitor and evaluate the growth rate and morphological changes of organoids.

[0154] c) System stability: Evaluate the stability and anti-interference ability of the control system.

[0155] (3) Establish a regular optimization mechanism: a) Regularly adjust model hyperparameters and control strategies based on performance evaluation results.

[0156] b) Design and execute small-scale exploratory experiments to discover potential optimization space without affecting daily experiments.

[0157] Through the above steps, the trained deep reinforcement learning model can be successfully deployed to the organoid culture system to achieve high-precision real-time control of drug concentration. This method can not only adapt to the complex and changing culture environment, but also dynamically adjust the control strategy according to the real-time physiological response of the organoid, significantly improving the accuracy and reliability of drug concentration control. At the same time, through online learning and continuous optimization mechanisms, the system can continuously improve its performance and provide a more stable and efficient experimental platform for organoid research and drug testing.

[0158] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0159] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for controlling drug concentration in organoid culture medium based on deep reinforcement learning, characterized in that: The process includes: Construct a drug concentration control model for organoid culture medium based on deep reinforcement learning, using the current drug concentration of the culture medium, organoid physiological response data, and environmental parameters as model inputs, the optimal drug concentration as the control target, and outputting the set value of the flow rate adjustment of the drug delivery pump; Initially train the drug concentration control model, then integrate the trained model into the system's control unit to receive and process sensor data in real time. Through real-time feedback, continuously compare the deviation between the actual drug concentration and the target concentration, and monitor the physiological response data of the organoids to dynamically adjust the control strategy. The physiological response data of organoids include growth rate and metabolic level; the growth rate objective function is defined, and the growth rate is controlled by controlling the drug concentration. Reach the maximum value while ensuring that the drug concentration is within the allowed range; set the metabolic level constraint function to ensure that the metabolic level of the organoid does not deviate from the baseline level by more than the set deviation , to ensure that the effect of drug concentration on metabolic levels is within an acceptable range.

2. The method for controlling drug concentration in organoid culture medium based on deep reinforcement learning according to claim 1, characterized in that: The original data set collection process for initial training of the drug concentration control model is as follows: Drug concentration data for organoid culture medium : Use liquid chromatography-mass spectrometry HPLC-MS to measure the drug concentration in the culture medium every few minutes and record the concentration at different time points Drug concentration , construct a time series dataset; Growth rate : Real-time cell analyzer RTCA was used to monitor the size changes of organoids and calculate the size of each time interval Relative growth rate: Metabolic level :Use fluorescence-based glucose and lactate metabolism sensors to monitor in real time, record glucose consumption rate and lactate production rate, and comprehensively calculate the metabolic level index; Multi-dimensional environmental parameters: including temperature, pH value, oxygen concentration, and record the corresponding relationship between these parameters and drug concentration: { , , } → All the collected original data sets are recorded as .

3. A method for controlling drug concentration in organoid culture medium based on deep reinforcement learning as claimed in claim 2, characterized in that: The original data set needs to be preprocessed, including data cleaning, data standardization, and label determination, and divided into training sets, validation sets, and test sets; The data cleaning includes outlier detection and processing using the Z-score method, noise removal using the Savitzky-Golay filter, and missing value filling; The data standardization is a normalization process; The label adjudication includes the definition of target drug concentration labels for the culture medium, the setting of organoid physiological response targets, and label calibration.

4. A method for controlling drug concentration in organoid culture medium based on deep reinforcement learning as claimed in claim 3, characterized in that: The culture medium target drug concentration label is specifically defined as: Determine the effect of drugs in organoid culture to set the optimal drug concentration range for each combination , the optimal concentration range represents the range in which the drug concentration can achieve the best effect in organoid culture, and the objective function is constructed as follows: in, is the actual drug concentration after normalization, is the optimal drug concentration, and are the maximum and minimum values ​​of the optimal concentration range respectively; this objective function measures the actual drug concentration With the optimal concentration The closer the value of the objective function is to 1, the closer the actual concentration is to the optimal concentration.

5. The method for controlling drug concentration in organoid culture medium based on deep reinforcement learning as claimed in claim 3, characterized in that: The organoid physiological response target formulation includes defining a growth rate objective function and setting a metabolic level constraint function; The growth rate objective function is defined as: in, For time The growth rate target at the moment, For in time The actual growth rate at that moment, For in time The drug concentration at the time, and are the minimum and maximum allowable drug concentrations respectively; the goal is to control the drug concentration to increase the growth rate to achieve the maximum value while ensuring that the drug concentration is within the permissible range; The metabolic level constraint function is set as follows: in, For time The metabolic level constraints at all times, For time Baseline metabolic level at the time, is the allowable deviation range.

6. The method for controlling drug concentration in organoid culture medium based on deep reinforcement learning according to claim 1, characterized in that: The organoid culture fluid drug concentration control model is a designed hybrid architecture, including an input module, a convolutional neural network module, a long short-term memory network module and an output module; The input module receives multi-dimensional data, including current drug concentration , Growth rate , metabolic level And environmental parameters, including temperature , pH value and oxygen concentration ; These input data are organized into a multidimensional tensor , expressed as: The convolutional neural network module is used for feature extraction, for extracting spatial features in input data, especially the complex relationship between environmental parameters and drug concentration; The long short-term memory network module is used for feature enhancement, effectively capturing and retaining long-term and short-term dependencies in time series data; The output module uses a reinforcement learning method to output the next action to be taken, that is, the set value of the flow rate adjustment amount of the drug delivery pump, according to the current state environment, including the current drug concentration, organoid growth rate, metabolic level and environmental parameters.

7. The method for controlling drug concentration in organoid culture medium based on deep reinforcement learning according to claim 6, characterized in that: The convolutional neural network module includes multiple convolutional layers and pooling layers; for the moment Tensor data , as non-sequential data for convolution operation: in, It is The characteristics of the layer, indicating the position in the layer The characteristic value of It is The convolution kernel of the layer is used to extract specific types of features. Represents the relative position of the elements in the convolution kernel, is the output of the previous layer, that is, the input data or the features of the previous layer, is a bias term used to adjust the convolution result. It is an activation function used to introduce nonlinear features and accelerate convergence; Through convolution operations, the convolutional neural network module can extract local features in the input data and further reduce the feature dimension and computational complexity through the pooling layer.

8. The method for controlling drug concentration in organoid culture medium based on deep reinforcement learning according to claim 6, characterized in that: The long short-term memory network module includes a forget gate, an input gate and an output gate; Forget Gate Determine the past information degree of retention; Input Gate and candidate cell states Determines the current input The importance of, and how to update cell state; Cell state The choice of forget gate and input gate are combined to form the main memory of the current time step; Output Gate Combined with the cell state, it determines the hidden state output for the next time step What is; The LSTM network module effectively captures and retains long-term and short-term dependencies in time series data through these gating mechanisms.

9. The method for controlling drug concentration in organoid culture medium based on deep reinforcement learning according to claim 6, characterized in that: The output module uses a deep Q network DQN and a reinforcement learning method to determine the next action to be taken, i.e., the set value adjustment of the flow rate adjustment of the drug delivery pump, according to the current state environment, i.e., the current drug concentration, organoid growth rate, metabolic level and environmental parameters, so as to achieve precise control of the drug concentration; DQN continuously iterates and updates the Q value function, and finally finds a strategy that maximizes the cumulative reward, where the Q value function is: in, Indicates the current time status, i.e., current drug concentration, organoid growth rate, metabolic levels, and environmental parameters; Indicates the current state The action taken is the set value adjustment amount of the drug delivery pump flow rate adjustment amount; Indicates the current time The reward obtained, i.e., the immediate response of the organoids to the drug concentration; is the discount factor, which is used to measure the importance of future returns. is the total number of iterative updates; Use the epsilon-greedy iteration strategy to The probability of choosing the action with the largest Q value is: The largest action selected Converting into specific control instructions takes into account the current state and target state of drug concentration, as well as the physical limitations of the system; The output module finally generates a normalized control instruction ,in: A negative value indicates a decrease in drug concentration; a positive value indicates an increase in drug concentration; the larger the absolute value, the greater the adjustment; Control instructions Through a mapping function Converted to actual drug delivery pump flow rate adjustment : in, is the maximum drug delivery pump flow rate allowed by the system, The function ensures that the output is within a reasonable range; Finally, the controller will It is sent as a control target instruction to achieve precise regulation of the drug concentration in the culture medium.

10. The method for controlling drug concentration in organoid culture medium based on deep reinforcement learning according to claim 1, characterized in that: In the process of learning the drug concentration control model of organoid culture medium, simulation experiments are combined with actual feedback; Build a simulation environment that can simulate the growth of different types of organoids under various conditions and the effects of changing drug concentrations on organoids; the model learns the optimal drug concentration control strategy by interacting with the environment; In actual organoid culture experiments, data are collected regularly, including drug concentrations, organoid growth conditions, and metabolic levels. These data are used to evaluate the performance of the model in the actual environment and serve as a basis for model updates; Use the collected actual experimental data to fine-tune the model online. Use the incremental learning method to update the model parameters using small batches of actual data. During the fine-tuning process, keep the knowledge learned by the model in the simulation environment while adapting to the characteristics of the actual environment. At the same time, domain adaptation technology is adopted and adversarial training methods are used to enable the model to learn to extract common features from simulated data and actual data, thereby improving the generalization ability of the model in actual environments.

Citation Information

Cited By

  • Single-cell drug sensitivity prediction algorithm based on multi-modal reinforcement learning

    CN119851746A

  • Organ-like biological sample live library management system based on artificial intelligence

    CN121053650A