Laboratory reagent bottle position dynamic adjustment and path planning method and system

Abstract

The invention discloses a laboratory reagent bottle position dynamic adjustment and path planning method and system. The reagent bottle position dynamic adjustment and path planning method comprises the steps that a laboratory environment state model is established, a deep Q network model is constructed and trained, and then path planning and dynamic optimization of a mechanical arm are conducted through the trained deep Q network model. A reward mechanism is set for the constructed deep Q network model, an experience playback mechanism is introduced, the shortest path and / or the minimum operation frequency are / is adopted as a target, Q value updating is carried out by adopting a strategy that a target network and a main network are separated, and parameters of the target network are updated once from the main network every fixed step number; in the model training process, taking and placing operation of the mechanical arm is simulated, and the strategy is updated according to an operation result. The method can quickly adapt to the conditions of reagent bottle position change, experimenter intervention, mechanical arm sudden failure and the like in a laboratory environment, and self-adaptive optimization of path planning is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of laboratory intelligent management, and more specifically, to a method and system for dynamic position adjustment and path planning of laboratory reagent bottles. Background Art

[0002] In modern laboratories, automated management of reagents faces challenges such as low management efficiency and unintelligent path planning. Traditional reagent management systems mostly rely on manual operation or semi-automated equipment, mainly relying on single-modal visual recognition technology, which has poor adaptability to complex environments such as lighting changes, reagent bottle occlusion and stacking, resulting in low recognition accuracy. These problems make manual intervention and adjustment an indispensable part of experimental operations, greatly reducing overall management efficiency.

[0003] Most of the existing automated robot path planning algorithms are based on static environment assumptions and lack the ability to dynamically respond to real-time environments. Therefore, during the operation, invalid actions, repeated paths, and poor planning often occur, resulting in low efficiency of robot operation.

[0004] At present, classic path planning algorithms (such as A* algorithm and Dijkstra algorithm) are usually applied to fixed path planning in a predetermined environment, but these methods cannot cope with real-time changes in the laboratory environment, such as reagent bottle position adjustment, experimenter intervention, and sudden failure of the robotic arm. This planning strategy lacks dynamic optimization capabilities, which makes the automation system perform poorly in complex environments, making it difficult to adjust the operation strategy and path planning in real time, seriously affecting the laboratory's operating efficiency.

[0005] Therefore, there is an urgent need to develop a method and system for dynamic adjustment of the position of laboratory reagent bottles and path planning to solve the problems of poor path planning and low operating efficiency of the robotic arm during the process of picking up and placing reagent bottles. Summary of the invention

[0006] In view of the problems existing in the prior art, the present invention proposes a method and system for dynamic adjustment of the position of laboratory reagent bottles and path planning. By combining environmental state modeling and reinforcement learning strategy optimization mechanism, it can quickly adapt to changes in the position of reagent bottles, intervention by experimenters and sudden failures of robotic arms in the laboratory environment, and realize adaptive optimization of path planning.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a method for dynamically adjusting the position of a laboratory reagent bottle and planning a path, comprising:

[0008] Establishing a laboratory environment state model: the environment state model includes but is not limited to a reagent rack, a reagent bottle operating robot arm, the real-time position of the reagent bottle, and the operating trajectory of the robot arm;

[0009] Build and train a deep Q network model, and then use the trained deep Q network model to perform path planning and dynamic optimization of the robot arm;

[0010] The deep Q network model is a deep reinforcement learning algorithm based on Q learning, which approximates the state-action value (Q value function) by introducing a deep neural network to perform strategy learning in a high-dimensional state space;

[0011] The input of the deep Q network model is the current state of the robot arm , the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robot arm is simulated, and the strategy is updated according to the operation results.

[0012] Furthermore, the environmental state model includes the following elements:

[0013] State Space:

[0014] State Space Represents all possible states of the robot in the laboratory environment; each state Described by the following characteristics:

[0015] Reagent rack layout information: indicates the position, type and placement status of each reagent bottle in the reagent rack;

[0016] The position and posture of the robot arm: describes the current spatial position of the robot arm (such as coordinates) and the rotation angle of its gripper;

[0017] The position and properties of the target reagent bottle: indicates the target position and related features of the reagent bottle that the robot currently needs to pick up and place;

[0018] Action Space:

[0019] Action Space Describe the robot arm in each state All operations that can be performed under, including:

[0020] The movement direction of the robot arm; the gripping action of the robot arm; the path adjustment of the robot arm;

[0021] Reward Function:

[0022] Reward Function Used to measure the robot arm's performance in performing a certain action After the performance, the reward strategy is as follows:

[0023] When successfully taking or placing the reagent bottle, positive rewards are given;

[0024] If a collision or drop occurs during the picking and placing process, a negative reward is given;

[0025] When the robot reaches the target position along the shortest path, a path efficiency reward is given.

[0026] Furthermore, the data of the state space and action space are derived from a multimodal perception module constructed on a robotic arm that operates reagent bottles: the multimodal perception module includes multiple sensors including RGB cameras, depth cameras, torque sensors, and temperature and humidity sensors; the data of the multiple sensors are aligned and synchronously processed through time and space coordinates to form complete environmental perception data at the same time point; in the data fusion process, a method combining Kalman filtering and particle filtering is used to correct the measurement errors of the multiple sensors; a deep learning visual recognition module based on convolutional neural networks and image segmentation technology is used to identify the types of reagent bottles and extract label information.

[0027] Furthermore, the Q value function of the deep Q network model is Indicates in status Next select action The expected cumulative reward that can be obtained; the update formula of the Q value function is as follows:

[0028]

[0029] in: The learning rate controls the amplitude of model parameter updates; Discount Factor is used to measure the impact of future rewards on current decisions, and its value range is [0, 1]; is the immediate reward of the current strategy, measuring the current action In Status The performance below; For the next state The maximum Q value of all actions in the state represents the maximum reward that can be obtained by choosing the best action in the next state.

[0030] Furthermore, the deep Q network model includes an input layer, a hidden layer network, and an output layer; the input layer is the current state The hidden layer network adopts a three-layer fully connected network, each layer contains several neurons, and uses the ReLU activation function for nonlinear mapping: ; The number of neurons in the output layer is equal to the action space The number of executable actions in the network, each neuron corresponds to the Q value of an action; through network forward propagation and Q value calculation, the model can Generate each action The Q value of the experience replay is used to select the optimal action; the experience replay mechanism includes the establishment of an experience replay pool, and randomly sampling a training data from the experience replay pool to update the Q value.

[0031] Furthermore, the training and strategy update of the deep Q network model includes the following processes:

[0032] Initialize Q network parameters: Randomly initialize the main network Parameters , and copy a copy to the target network ;

[0033] Let the robot interact in a laboratory environment and choose actions according to the ε-greedy policy: The probability of randomly selecting an action is The probability of selecting the action corresponding to the maximum Q value is gradually reduced as the training progresses. The value of

[0034] Q value update and training:

[0035] Randomly sample mini-batches of data from the experience replay pool, calculate the target Q value for each sample, and minimize the following loss function:

[0036]

[0037] Among them, the target Q value The calculation is as follows:

[0038]

[0039] and Represent the parameters of the main network and the target network respectively.

[0040] Policy update and target network update:

[0041] Every fixed number of steps, the parameters of the main network are adjusted Copy to target network and update the current policy.

[0042] Furthermore, the experience replay pool capacity is set to 10,000, storing the most recent 10,000 interaction data.

[0043] Furthermore, the update frequency of the target network is set to 1000 steps, and the parameters of the main network are copied to the target network every 1000 steps.

[0044] In a second aspect, the present invention provides a laboratory reagent bottle position dynamic adjustment and path planning method system, which is used to implement the laboratory reagent bottle position dynamic adjustment and path planning method as described above, comprising:

[0045] The laboratory environment state model includes but is not limited to the reagent rack, the reagent bottle operating robot, the real-time position of the reagent bottle and the operating trajectory of the robot;

[0046] The deep Q network model and its training module, based on the deep reinforcement learning algorithm of Q learning, approximates the state-action value (Q value function) by introducing a deep neural network to perform path planning and dynamic optimization of the robot arm:

[0047] The input of the deep Q network model is the current state of the robot arm , the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robot arm is simulated, and the strategy is updated according to the operation results.

[0048] Furthermore, the laboratory environment status model tracks and records information including the frequency of use, expiration date, storage location and environmental conditions of reagent bottles in real time, and uploads it to the laboratory management central controller to guide the layout optimization of the reagent bottle locations.

[0049] Compared with the prior art, the present invention has the following technical effects:

[0050] 1. Reinforcement learning strategy optimization: The present invention adopts the deep reinforcement learning (DRL) algorithm, which can dynamically adjust the pick-and-place strategy according to the real-time state changes of the laboratory environment. Compared with the traditional static path planning algorithm (such as A*, Dijkstra algorithm), the present invention can quickly adapt to the position changes of reagent bottles, intervention of experimenters and sudden failure of the robot arm in the laboratory environment, realize the adaptive optimization of path planning, effectively avoid invalid operations and path redundancy, and greatly improve the operating efficiency of the system.

[0051] 2. Intelligent path planning and operation optimization: The present invention can automatically generate the optimal path and operation strategy by combining environmental state modeling with a strategy optimization mechanism. Unlike the fixed path planning in the existing methods, the present invention introduces a reinforcement learning model in real-time path planning, which can not only avoid obstacles, but also dynamically select the optimal operation path, reducing the idle time and repeated operations of the robot arm. The introduction of this module makes the system have a shorter path and more streamlined movements during the process of taking and placing reagent bottles, thereby significantly improving the operating efficiency and safety of the robot arm.

[0052] 3. Reagent status tracking and layout optimization: The present invention can dynamically record the storage location, usage frequency, expiration date and other information of all reagent bottles, guide intelligent layout optimization, and automatically adjust the placement of reagent bottles in the reagent rack. Compared with the traditional reagent management method that relies on manual or semi-automatic equipment, the present invention can help intelligently adjust the layout of reagent bottles, preferentially place high-frequency reagents in areas that are easy to access, and remind and process reagent bottles that are about to expire, thereby improving the overall intelligent level of management. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a flowchart of a method for dynamically adjusting the position of a laboratory reagent bottle and planning a path in one embodiment of the present invention.

[0054] Figure 2 FIG. 4 is an architecture diagram of a deep Q network model in one embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0056] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that well-known algorithms are not shown in detail to avoid obscuring the subject matter of the present invention.

[0057] In addition, the order of execution of actions, steps, etc. in the devices and methods shown in the claims, specifications and drawings can be implemented in any order as long as there is no special explicit limitation on the order and the output of the previous processing is not used in the subsequent processing.

[0058] Example 1

[0059] See also Figure 1 This embodiment provides a method for dynamically adjusting the position of a laboratory reagent bottle and planning a path, including:

[0060] Establishing a laboratory environment state model: The environment state model includes but is not limited to a reagent rack, a reagent bottle operating robot arm, the real-time position of the reagent bottle, and the operating trajectory of the robot arm.

[0061] Specifically, the environmental state model includes the following elements:

[0062] State Space:

[0063] State Space Represents all possible states of the robot in the laboratory environment. Each state Described by the following characteristics:

[0064] Reagent rack layout information: indicates the position, type and placement status of each reagent bottle in the reagent rack.

[0065] The position and posture of the robot arm: describes the current spatial position of the robot arm (such as coordinates) and the rotation angle of its gripper.

[0066] The position and properties of the target reagent bottle: indicates the target position and related characteristics (such as label information, weight, etc.) of the reagent bottle that the robot currently needs to pick up and place.

[0067] Action Space:

[0068] Action Space Describes the robot arm in each state All operations that can be performed under. Specifically include:

[0069] The direction of movement of the robot arm (such as up and down, left and right, front and back, etc.).

[0070] The gripping action of the robot arm (such as grasping, releasing, adjusting force, etc.).

[0071] Path adjustment of the robot arm (such as translation, rotation, scaling, etc.).

[0072] Reward Function:

[0073] Reward Function Used to measure the robot arm's performance in performing a certain action The specific reward strategy is as follows:

[0074] When the reagent bottle is successfully placed or retrieved, a positive reward (such as +10) is given.

[0075] If a collision or drop occurs during the picking and placing process, a negative reward (such as -20) is given.

[0076] When the robot reaches the target position along the shortest path, a path efficiency bonus (such as +5) is given.

[0077] Through the above definitions of state space, action space, and reward function, the system is able to model and evaluate the operation of the robotic arm in the laboratory.

[0078] Preferably, the data of the state space comes from a multimodal perception module constructed on a robotic arm that operates the reagent bottle: the multimodal perception module includes a variety of sensors including an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor; the RGB camera is used to capture the color, label information and surface morphology characteristics of the reagent bottle, and generate two-dimensional image data; the depth camera is used to collect the depth information and relative distance of the reagent bottle in three-dimensional space; the torque sensor is used to monitor the contact force of the robotic arm when grasping and moving the reagent bottle to ensure the safety of the operation process; the temperature and humidity sensor monitors the temperature and humidity changes in the laboratory environment in real time to ensure the environmental stability of the reagent during storage.

[0079] The data of the multiple sensors are aligned and synchronously processed through time and space coordinates to form complete environmental perception data at the same time point; during the data fusion process, the measurement errors of the multiple sensors are corrected by combining Kalman filtering and particle filtering.

[0080] This embodiment adopts a multimodal data fusion method combining Kalman filtering and particle filtering to reduce the measurement errors between sensors and effectively improve the accuracy and robustness of environmental perception.

[0081] Kalman filtering is a recursive optimal state estimation method commonly used in linear systems and is suitable for processing linear sensor data with Gaussian noise. This embodiment uses Kalman filtering to fuse the data of the torque sensor and the temperature and humidity sensor to eliminate measurement noise and improve data accuracy. Specifically, Kalman filtering is divided into two stages: prediction stage and update stage.

[0082] Prediction stage: Based on the state at the previous moment and the current control input, predict the state and state covariance matrix at the current moment:

[0083]

[0084]

[0085] in Indicates time The predicted status. is the state transfer matrix, which indicates the system from the previous moment To the current time The state transition relationship. B is the control matrix, which represents the control input Impact on system status. is the covariance matrix of the predicted state. is the process noise covariance matrix.

[0086] Update phase: When new sensor measurements are When it arrives, the state is updated according to the predicted state and the observed value:

[0087]

[0088]

[0089]

[0090] in: is the Kalman gain, which represents the weight distribution between the sensor measurement value and the predicted state, is the observation matrix, which represents the mapping relationship from state space to observation space. is the measurement noise covariance matrix. is the updated optimal state estimate. Kalman filtering can effectively reduce noise and make prediction corrections to linear sensor data, thereby improving the measurement accuracy of torque sensors and temperature and humidity sensors.

[0091] Particle filtering is a nonlinear state estimation technology based on the Monte Carlo method, which is suitable for processing complex data with nonlinear and non-Gaussian distribution. This embodiment uses particle filtering to fuse the visual data of the RGB camera and the depth camera to process the three-dimensional position information in a complex environment.

[0092] The particle filter represents the probability distribution of the state through a set of discrete particles and their weights, and resamples them according to the weight of each particle. The weight update steps of the particle are as follows:

[0093] Particle state prediction: Assume that the particle state at the previous moment is , predict the particle state at the current moment:

[0094]

[0095] Where f is the state transfer function, is the process noise.

[0096] Particle weight update: based on current observation value and the particle's predicted state , update the particle weight:

[0097]

[0098] in is the likelihood probability between the particle state and the observed value, which can be estimated by Gaussian distribution.

[0099] Particle resampling: Resample the particle set according to the particle weight distribution, retain the particles with higher weights, and remove the particles with lower weights to avoid particle degradation. State estimation: The final state estimation value is the weighted average of all particle states:

[0100]

[0101] in is the particle weight, The state of the particle.

[0102] By fusing visual data with particle filtering, it is possible to maintain a high-precision estimate of the position of reagent bottles in complex environments (such as changes in the position of reagent bottles, occlusion, stacking, etc.), thereby improving the robustness and recognition capabilities of the system in complex environments.

[0103] Kalman filtering and particle filtering are used together in the present invention to form an efficient and accurate multimodal data fusion method. The specific steps are as follows:

[0104] Linear data processing: Use Kalman filtering to reduce noise and predict and update linear sensor data (such as torque sensors and temperature and humidity sensors) to improve the accuracy and stability of measurement data.

[0105] Nonlinear data processing: Use particle filtering to predict and estimate the state of nonlinear sensor data (such as RGB cameras and depth cameras), and update the weights of particles according to the current observation values ​​to ensure the fusion accuracy of visual information and 3D depth information.

[0106] Data fusion and joint estimation: The linear data processed by Kalman filter and the nonlinear data processed by particle filter are jointly fused. By weighted averaging, the information from different sensors is integrated into the state estimation at the same time, thus forming the complete state information of the reagent bottle. Assume that the linear state estimation is , the nonlinear state is estimated as , then the final state is estimated to be:

[0107]

[0108] in, It is the weight factor for linear and nonlinear data fusion, which is dynamically adjusted according to the complexity of the laboratory environment and the noise level of the sensor to ensure that the fused state estimation can maintain high robustness in various complex environments.

[0109] Adaptive dynamic adjustment mechanism: This embodiment preferably designs an adaptive dynamic adjustment mechanism to dynamically adjust the fusion ratio of linear filtering and nonlinear filtering according to the measurement accuracy and reliability of each sensor in different scenarios. For example, when the ambient lighting conditions are good and the label is not blocked, the system will increase the weight of visual recognition data; when the lighting conditions are poor or the reagent bottle is blocked, the weight of the torque and depth information will be increased to ensure accurate perception of the reagent bottle status.

[0110] Data anomaly detection and correction: In the data fusion process, this embodiment preferably introduces a data anomaly detection and correction mechanism. The specific approach is to perform residual analysis on the measurement values ​​output by each sensor and the fused state estimate. If the measurement residual of a sensor exceeds the set threshold, the sensor output is considered abnormal. At this time, the system will reduce the weight of the sensor data until its output returns to normal. This mechanism can effectively prevent overall perception failure caused by abnormal data from a single sensor.

[0111] In addition, a deep learning visual recognition module based on convolutional neural networks and image segmentation technology can be used to process images taken by the camera to identify the type of reagent bottles and extract label information.

[0112] The multimodal perception module, visual recognition module and data fusion thereof can realize high-precision recognition, positioning and environmental status perception of reagent bottles in a complex laboratory environment, and provide reliable data support for reagent management and operation.

[0113] On the basis of establishing a laboratory environment state model, this embodiment constructs and trains a deep Q-network (DQN) model, and then uses the trained deep Q-network model to perform path planning and dynamic optimization of the robotic arm.

[0114] The deep Q network model is a deep reinforcement learning algorithm based on Q learning, which approximates the state-action value (Q value function) by introducing a deep neural network to perform strategy learning in a high-dimensional state space;

[0115] The input of the deep Q network model is the current state of the robotic arm, and the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robotic arm is simulated, and the strategy is updated according to the operation results.

[0116] More specifically, the basic principles of DQN:

[0117] DQN approximates the Q-value function through a deep neural network (usually a multi-layer convolutional network or a fully connected network). Indicates in status Next select action The expected cumulative reward that can be obtained. The update formula of the Q value function is as follows:

[0118]

[0119] in: is the learning rate, which controls the amplitude of model parameter update. Discount Factor is used to measure the impact of future rewards on current decisions, and its value range is [0, 1]. is the immediate reward of the current strategy, measuring the current action In Status The following performance. For the next state The maximum Q value of all actions in the state represents the maximum reward that can be obtained by choosing the best action in the next state.

[0120] like Figure 2 As shown in the figure, the network structure of DQN consists of the following key links:

[0121] In the present invention, the input of the DQN network is the current state of the robot , the output is the Q value of all possible actions. env specifically refers to the laboratory environment state model composed of state space and action space information. The specific network structure is as follows:

[0122] The input layer is the current state The feature vector contains the position of the robot arm, the target position of the reagent bottle, and the relevant information of the laboratory environment.

[0123] The hidden layer network uses a three-layer fully connected network (Fully Connected Network), each layer contains 128 neurons, and uses the ReLU activation function for nonlinear mapping:

[0124]

[0125] The number of neurons in the output layer is equal to the action space The number of executable actions in the network, each neuron corresponds to the Q value of an action. Through network forward propagation and Q value calculation, the model can Generate each action The Q value of the action is used to select the optimal action.

[0126] Q value update and experience replay:

[0127] In order to improve the stability and efficiency of model training, this paper introduces an experience replay mechanism. The specific steps are as follows:

[0128] Establishment of experience replay pool:

[0129] During training, the model converts the quadruple of each interaction into Stored in the experience replay pool. The size of the experience replay pool is set to 10,000, which can store the most recent 10,000 interaction experiences.

[0130] Randomly sample training data:

[0131] During each training, a small batch (Batch Size is usually set to 32) of data is randomly sampled from the experience replay pool to update the Q value. This can break the temporal correlation between the training data and improve the generalization ability of the model.

[0132] Target Network is separated from Main Network:

[0133] DQN uses the target network and the main network The parameters of the target network are updated from the main network once every fixed number of steps (preferably 1000 times) to avoid frequent oscillations of the Q-value function during training.

[0134] Training and strategy updates:

[0135] Initialize the Q network parameters:

[0136] Randomly initialize the main network Parameters , and copy a copy to the target network .

[0137] Interaction and experience storage:

[0138] Let the robot interact in a laboratory environment and choose actions according to the ε-greedy policy: The probability of randomly selecting actions (exploration) is The probability of selecting the action corresponding to the maximum Q value (utilization). As the training progresses, it gradually decreases The value of (e.g., from 1 to 0.1) is reduced to increase the probability of using the optimal strategy.

[0139] Q value update and training:

[0140] Randomly sample mini-batches of data from the experience replay pool, calculate the target Q value for each sample, and minimize the following loss function:

[0141]

[0142] Among them, the target Q value The calculation is as follows:

[0143]

[0144] and Represent the parameters of the main network and the target network respectively.

[0145] Policy update and target network update:

[0146] Every fixed number of steps, the parameters of the main network are adjusted Copy to target network and update the current policy.

[0147] Through the above steps, the DQN model can adaptively generate the optimal operation strategy through continuous interaction and learning in a complex environment, and realize dynamic path planning and pick-and-place optimization of the robot arm.

[0148] DQN network training:

[0149] In order to verify the effectiveness of the dynamic path planning module based on multimodal perception, deep learning visual recognition and deep reinforcement learning in the present invention, we conducted detailed model training and experimental testing. The following is the specific training process and experimental settings.

[0150] Model training parameters:

[0151] Learning Rate (α): Set to 0.001 to control the learning speed of the model and ensure that the model can gradually converge during the gradient descent process.

[0152] Discount Factor (γ): Set to 0.95, indicating the weight ratio of future rewards. This parameter is used to measure the impact of long-term rewards on current decisions, so that the model pays more attention to long-term benefits when planning paths.

[0153] Exploration Rate (ε): Initial value is 1, which is gradually linearly decayed to 0.1. Use the ε-greedy strategy to randomly select actions with probability ε to explore new paths, select the best action with probability 1−ε, and gradually switch to using the learned strategy.

[0154] Experience replay pool capacity: set to 10,000, storing the most recent 10,000 interaction data to ensure sample diversity and break the temporal correlation between data.

[0155] Batch Size: Set to 32. 32 samples are randomly selected from the experience replay pool for small batch training each time.

[0156] Target network update frequency: set to 1000 steps, copy the parameters of the main network to the target network to improve the stability of Q value update.

[0157] Training environment:

[0158] The training was performed on an NVIDIA GeForce RTX 4060 GPU to increase the training speed, equipped with an Intel i7-14700HX CPU and 32GB of memory, with Python as the programming language and Pytorch as the deep learning framework. Parallel computing was used to speed up the training process of the model on large-scale data sets, allowing it to be trained in a reasonable time.

[0159] Data integration and training process:

[0160] In order to utilize the data of the multimodal perception module and the deep learning visual recognition module, the following data processing and integration methods are used during training:

[0161] State input: The RGB image, depth information, torque data, and temperature and humidity data collected by the multimodal perception module are pre-processed and input into the DQN model as a representation of the robot's environmental state. These data can help the model better perceive the laboratory environment and the specific location of the reagent bottle.

[0162] Visual data fusion: The reagent bottle label information (such as color, shape, etc.) output by the deep learning visual recognition module is used as an additional state feature to help DQN more accurately determine the type of target reagent bottle and its optimal operation strategy.

[0163] Interaction and experience playback: The state of each robot operation (including success rate, path length, operation time, etc.) is recorded as a four-tuple The samples saved in the experience replay pool are randomly sampled and used for subsequent training to improve data utilization.

[0164] The training of DQN is achieved through the following steps:

[0165] Initialize Q network parameters: Randomly initialize the parameters of the main network Q and copy them to the target network to ensure the stability of the initial Q value.

[0166] Exploration and exploitation strategy: Use a high ε value (ε = 1) at the beginning of training, conduct a lot of exploration to accumulate diverse operation data; as training progresses, gradually decay the ε value and gradually turn to exploiting the current strategy.

[0167] Loss function optimization: Randomly draw samples from the experience replay pool, calculate the target Q value for each sample, and update the network weights by minimizing the following loss function:

[0168]

[0169] Among them, the target Q value , and Represent the parameters of the main network and the target network respectively.

[0170] Target network update: Every 1,000 steps, the parameters of the main network are copied to the target network to enhance stability during training.

[0171] After the DQN model training was completed, we verified the model through different indicators:

[0172] Path optimization effect: The path generated by the DQN model after training is about 30% shorter than the random path, indicating that the model can select the optimal path more intelligently.

[0173] Operation success rate: After testing on multiple simulated reagent rack layouts, the model's pick-and-place task success rate reached 96%, indicating that under most experimental conditions, the model can accurately complete the operation of reagent bottles.

[0174] Improved efficiency: The average operation time of the model in completing the pick-and-place task was reduced by 25%, indicating that after training with multimodal perception and visual recognition data, the DQN model has a more comprehensive understanding of the laboratory environment, thereby effectively improving operational efficiency.

[0175] Through the dynamic update of the laboratory environment state model, the DQN model can be continuously optimized and improved during training, making the path planning of the robot arm in a complex laboratory environment more intelligent and accurate, thereby improving the overall operational efficiency and reliability of the system. These training results demonstrate the potential of the present invention in practical applications and provide an advanced solution for intelligent laboratory management.

[0176] Example 2

[0177] This embodiment provides a laboratory reagent bottle position dynamic adjustment and path planning method system, which is used to implement the laboratory reagent bottle position dynamic adjustment and path planning method as described in Example 1, including:

[0178] The laboratory environment state model includes but is not limited to the reagent rack, the reagent bottle operating robot, the real-time position of the reagent bottle and the operating trajectory of the robot;

[0179] The deep Q network model and its training module, based on the deep reinforcement learning algorithm of Q learning, approximates the state-action value (Q value function) by introducing a deep neural network to perform path planning and dynamic optimization of the robot arm:

[0180] The input of the deep Q network model is the current state of the robot arm , the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robot arm is simulated, and the strategy is updated according to the operation results.

[0181] As a preferred technical solution, the laboratory environment status model also tracks and records information including the frequency of use, validity period, storage location and environmental conditions of reagent bottles in real time, and uploads it to the laboratory management central controller. The central controller generates management instructions (such as robot arm pick-up and placement path planning, storage environment adjustment, alarm prompts, etc.) based on the status information, and sends them to the corresponding execution module. At the same time, the central controller can combine the optimal path results of the path planning system to globally optimize the reagent rack layout to ensure that the robot arm has the shortest path and the least operation time when performing the pick-up and placement tasks.

[0182] The above-mentioned laboratory reagent bottle position dynamic adjustment and path planning method can be embodied in the form of a computer program product or a software functional unit. If the above-mentioned laboratory reagent bottle position dynamic adjustment and path planning method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, the technical solution is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for an electronic system (which can be a personal computer, a server, or a network system, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0183] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0184] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0185] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the systems and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A method for dynamic position adjustment and path planning of laboratory reagent bottles, characterized in that: include: Establishing a laboratory environment state model: the environment state model includes but is not limited to a reagent rack, a reagent bottle operating robot arm, the real-time position of the reagent bottle, and the operating trajectory of the robot arm; Build and train a deep Q network model, and then use the trained deep Q network model to perform path planning and dynamic optimization of the robot arm; The deep Q network model is a deep reinforcement learning algorithm based on Q learning, which approximates the state-action value (Q value function) by introducing a deep neural network to perform strategy learning in a high-dimensional state space; The input of the deep Q network model is the current state of the robot arm , the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robot arm is simulated, and the strategy is updated according to the operation results.

2. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 1, characterized in that: The environmental state model includes the following elements: State Space: State Space Represents all possible states of the robot in the laboratory environment; each state Described by the following characteristics: Reagent rack layout information: indicates the position, type and placement status of each reagent bottle in the reagent rack; The position and posture of the robot arm: describes the current spatial position of the robot arm (such as coordinates) and the rotation angle of its gripper; The position and properties of the target reagent bottle: indicates the target position and related features of the reagent bottle that the robot currently needs to pick up and place; Action Space: Action Space Describe the robot in each state All operations that can be performed under, including: The movement direction of the robot arm; the gripping action of the robot arm; the path adjustment of the robot arm; Reward Function: Reward Function Used to measure the robot arm's performance in performing a certain action After the performance, the reward strategy is as follows: When successfully taking or placing the reagent bottle, give positive rewards; If a collision or drop occurs during the picking and placing process, a negative reward is given; When the robot reaches the target position along the shortest path, a path efficiency reward is given.

3. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 2, characterized in that: The data of the state space and action space are derived from a multimodal perception module constructed on the robotic arm that operates the reagent bottle: the multimodal perception module includes multiple sensors including an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor; the data of the multiple sensors are processed by aligning and synchronizing the time and space coordinates to form complete environmental perception data at the same time point; During the data fusion process, a method combining Kalman filtering and particle filtering is used to correct the measurement errors of the multiple sensors; a deep learning visual recognition module based on convolutional neural network and image segmentation technology is used to identify the types of reagent bottles and extract label information.

4. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 2, characterized in that: Q value function of the deep Q network model Indicates in status Next select action The expected cumulative reward that can be obtained; the update formula of the Q value function is as follows: in: The learning rate controls the amplitude of model parameter updates; Discount Factor is used to measure the impact of future rewards on current decisions, and its value range is [0, 1]; is the immediate reward of the current strategy, measuring the current action In Status The performance below; For the next state The maximum Q value of all actions in the state represents the maximum reward that can be obtained by choosing the best action in the next state.

5. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 4, characterized in that: The deep Q network model includes an input layer, a hidden layer network, and an output layer; the input layer is the current state The hidden layer network adopts a three-layer fully connected network, each layer contains several neurons, and uses the ReLU activation function for nonlinear mapping: ; The number of neurons in the output layer is equal to the action space The number of executable actions in the network, each neuron corresponds to the Q value of an action; through network forward propagation and Q value calculation, the model can Generate each action The Q value of the experience replay is used to select the optimal action; the experience replay mechanism includes the establishment of an experience replay pool, and randomly sampling a training data from the experience replay pool to update the Q value.

6. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 5, characterized in that: The training and strategy update of the deep Q network model includes the following processes: Initialize Q network parameters: Randomly initialize the main network Parameters , and copy a copy to the target network ; Let the robot interact in a laboratory environment and choose actions according to the ε-greedy policy: The probability of randomly selecting an action is The probability of selecting the action corresponding to the maximum Q value is gradually reduced as the training progresses. The value of Q value update and training: Randomly sample mini-batches of data from the experience replay pool, calculate the target Q value for each sample, and minimize the following loss function: Among them, the target Q value The calculation is as follows: and Represent the parameters of the main network and the target network respectively. Policy update and target network update: Every fixed number of steps, the parameters of the main network are adjusted Copy to target network and update the current policy.

7. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 6, characterized in that: The experience replay pool capacity is set to 10,000, storing the most recent 10,000 interaction data.

8. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 6 or 7, characterized in that: The update frequency of the target network is set to 1000 steps, and the parameters of the main network are copied to the target network every 1000 steps.

9. Laboratory reagent bottle position dynamic adjustment and path planning method system, characterized in that: A method for dynamically adjusting the position of a laboratory reagent bottle and planning a path according to any one of claims 1 to 8, comprising: The laboratory environment state model includes but is not limited to the reagent rack, the reagent bottle operating robot, the real-time position of the reagent bottle and the operating trajectory of the robot; The deep Q network model and its training module, based on the deep reinforcement learning algorithm of Q learning, approximates the state-action value (Q value function) by introducing a deep neural network to perform path planning and dynamic optimization of the robot arm: The input of the deep Q network model is the current state of the robot arm , the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robot arm is simulated, and the strategy is updated according to the operation results.

10. The laboratory reagent bottle position dynamic adjustment and path planning system according to claim 9, characterized in that: The laboratory environment status model tracks and records information including the frequency of use, expiration date, storage location and environmental conditions of reagent bottles in real time, and uploads it to the laboratory management central controller.

Images