Laboratory reagent bottle position dynamic adjustment and path planning method and system

Abstract

The application discloses a laboratory reagent bottle position dynamic adjustment and path planning method and system. The reagent bottle position dynamic adjustment and path planning method comprises the following steps: establishing a laboratory environment state model, constructing and training a deep Q network model, and then adopting the trained deep Q network model to perform path planning and dynamic optimization of a mechanical arm. The constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts a shortest path and / or least operation number as a target, and adopts a target network and a main network separation strategy to perform Q value updating, and the parameters of the target network are updated from the main network once every fixed number of steps; in the model training process, the taking and placing operations of the mechanical arm are simulated, and the strategy is updated according to the operation result. The application can quickly adapt to reagent bottle position changes, experimental personnel intervention and mechanical arm sudden failures and the like in a laboratory environment, and realizes adaptive optimization of path planning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of laboratory intelligent management, more particularly, to a laboratory reagent bottle position dynamic adjustment and path planning method and system. BACKGROUND

[0002] In modern laboratories, the automation management of reagents faces challenges such as low management efficiency and unintelligent path planning. Traditional reagent management systems mostly rely on manual operation or semi-automated equipment, mainly relying on single-modal visual recognition technology, which has poor adaptability to complex environments such as changes in light, reagent bottle occlusion, and stacking, resulting in low recognition accuracy. These problems make manual intervention and adjustment an essential part of experimental operations, significantly reducing overall management efficiency.

[0003] Existing automated robotic arm path planning algorithms are mostly based on static environment assumptions and lack dynamic response capabilities to real-time environments. Therefore, during operation, there are often invalid actions, repeated paths, and poor planning, resulting in low robotic arm operation efficiency.

[0004] Currently, classic path planning algorithms (such as A* algorithm, Dijkstra algorithm) are usually applied to fixed path planning in predetermined environments, but these methods cannot cope with real-time changes in laboratory environments, such as reagent bottle position adjustment, experimental personnel intervention, and robotic arm sudden failure. This lack of dynamic optimization capability in planning strategies makes the automation system perform poorly in complex environments, making it difficult to adjust operation strategies and path planning in real time, severely affecting laboratory operation efficiency.

[0005] Therefore, there is an urgent need to develop a laboratory reagent bottle position dynamic adjustment and path planning method and system to solve the problems of poor path planning and low operation efficiency of the robotic arm during reagent bottle picking and placing. SUMMARY

[0006] Due to the problems of existing technologies, the present application proposes a laboratory reagent bottle position dynamic adjustment and path planning method and system, which can quickly adapt to changes in reagent bottle positions, experimental personnel intervention, and robotic arm sudden failure in laboratory environments by combining environment state modeling and reinforcement learning strategy optimization mechanisms, achieving adaptive optimization of path planning.

[0007] To achieve the above-mentioned purposes, in a first aspect, the present application provides a laboratory reagent bottle position dynamic adjustment and path planning method, comprising:

[0008] Establishing a laboratory environment state model: the environment state model includes but is not limited to reagent racks, reagent bottle operation robotic arms, real-time positions of reagent bottles, and operation trajectories of the robotic arms;

[0009] The deep Q network model is constructed and trained, and then the trained deep Q network model is used for path planning and dynamic optimization of the robot arm.

[0010] The deep Q network model is a deep reinforcement learning algorithm based on Q learning, which approximates the state-action value (Q value function) by introducing a deep neural network, and performs policy learning in a high-dimensional state space.

[0011] The input of the deep Q network model is the current state of the robot arm , and the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, uses the shortest path and / or the least number of operations as the target, and uses a target network and a main network separation strategy to update the Q value, and the parameters of the target network are updated from the main network every fixed number of steps; during the training process of the deep Q network model, the picking and placing operations of the robot arm are simulated, and the policy is updated according to the operation results.

[0012] Further, the environment state model includes the following elements:

[0013] State space (State Space):

[0014] State space represents all possible states of the robot arm in the laboratory environment; each state is described by the following features:

[0015] Layout information of reagent rack: indicates the position, type and placement state of each reagent bottle in the reagent rack;

[0016] Position and posture of robot arm: describes the current spatial position (such as coordinates) of the robot arm and the rotation angle of its gripper;

[0017] Position and attributes of target reagent bottle: indicate the target position and related features of the reagent bottle that needs to be picked and placed by the robot arm;

[0018] Action space (Action Space):

[0019] Action space describes all operations that can be performed by the robot arm in each state , including:

[0020] Moving direction of robot arm; gripping action of robot arm; path adjustment of robot arm;

[0021] Reward function (Reward Function):

[0022] Reward function The reward policy is as follows:

[0023] A positive reward is given when the reagent bottle is successfully picked and placed.

[0024] A negative reward is given when a collision or drop occurs during the picking and placing process.

[0025] A path efficiency reward is given when the robot arm reaches the target position along the shortest path.

[0026] Further, the data of the state space and action space are derived from a multi-modal perception module constructed on the robot arm operating the reagent bottle, which includes multiple sensors such as an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor. The data of the multiple sensors are aligned and synchronized through time and space coordinates to form complete environmental perception data at the same time point. During data fusion, Kalman filtering and particle filtering are combined to correct the measurement errors of the multiple sensors. A deep learning visual recognition module based on convolutional neural networks and image segmentation technology is used for reagent bottle type identification and label information extraction.

[0027] Further, the Q value function of the deep Q network model is which represents the expected cumulative reward that can be obtained by selecting action in state The update formula of the Q value function is as follows:

[0028]

[0029] Wherein: is the learning rate (Learning Rate) that controls the magnitude of model parameter update; is the discount factor (Discount Factor) that measures the degree of influence of future rewards on current decisions, with a value range of [0, 1]; is the immediate reward (Reward) of the current policy, which measures the performance of the current action in state ; is the maximum Q value of all actions in the next state , which represents the maximum return that can be obtained by selecting the optimal action in the next state.

[0030] Further, the deep Q network model includes an input layer, a hidden layer network, and an output layer. The input layer is a feature vector of the current state . The hidden layer network uses a three-layer fully connected network, each layer containing a number of neurons and using a ReLU activation function for non-linear mapping:​ ; The number of neurons in the output layer is equal to the action space The number of executable actions in the network, each neuron corresponds to the Q value of an action; through network forward propagation and Q value calculation, the model can Generate each action The Q value of the experience replay is used to select the optimal action; the experience replay mechanism includes the establishment of an experience replay pool, and randomly sampling a training data from the experience replay pool to update the Q value.

[0031] Furthermore, the training and strategy update of the deep Q network model includes the following processes:

[0032] Initialize Q network parameters: Randomly initialize the main network Parameters , and copy a copy to the target network ;

[0033] Let the robot arm interact in the laboratory environment and choose actions according to the ε-greedy policy: The probability of randomly selecting an action is The probability of selecting the action corresponding to the maximum Q value is gradually reduced as the training progresses. The value of

[0034] Q value update and training:

[0035] Randomly sample mini-batches of data from the experience replay pool, calculate the target Q value for each sample, and minimize the following loss function:

[0036]

[0037] Among them, the target Q value The calculation is as follows:

[0038]

[0039] and Represent the parameters of the main network and the target network respectively.

[0040] Policy update and target network update:

[0041] Every fixed number of steps, the parameters of the main network are adjusted Copy to target network and update the current policy.

[0042] Furthermore, the experience replay pool capacity is set to 10,000, storing the most recent 10,000 interaction data.

[0043] Further, the update frequency of the target network is set to 1000 steps, and the parameters of the main network are copied to the target network every 1000 steps.

[0044] In a second aspect, the application provides a laboratory reagent bottle position dynamic adjustment and path planning method system for implementing the laboratory reagent bottle position dynamic adjustment and path planning method as described above, comprising:

[0045] A laboratory environment state model, including but not limited to reagent racks, reagent bottle operation mechanical arms, real-time positions of reagent bottles, and operation trajectories of the mechanical arms;

[0046] A deep Q network model and its training module, a deep reinforcement learning algorithm based on Q learning, which approximates the state-action value (Q value function) by introducing a deep neural network, plans and dynamically optimizes the path of the mechanical arm:

[0047] The input of the deep Q network model is the current state of the mechanical arm , and the output is the Q value of all possible actions; the deep Q network model is constructed to set a reward mechanism, introduce an experience replay mechanism, use the shortest path and / or the least number of operations as the target, and use the target network and the main network separation strategy to update the Q value, and the parameters of the target network are updated from the main network every fixed number of steps; in the training process of the deep Q network model, the taking and placing operations of the mechanical arm are simulated, and the strategy is updated according to the operation results.

[0048] Further, the laboratory environment state model tracks and records information including the usage frequency, expiration date, storage location, and environmental conditions of the reagent bottles in real time, and uploads them to the laboratory management central controller to guide the layout optimization of the reagent bottle positions.

[0049] Compared with the prior art, the application has the following technical effects:

[0050] 1. Reinforcement learning strategy optimization: The application uses a deep reinforcement learning (DRL) algorithm, which can dynamically adjust the taking and placing strategy according to the real-time state changes of the laboratory environment. Compared with traditional static path planning algorithms (such as A* and Dijkstra algorithms), the application can quickly adapt to changes in reagent bottle positions, experimental personnel intervention, and mechanical arm failures in the laboratory environment, realize adaptive optimization of path planning, effectively avoid invalid operations and path redundancy, and greatly improve the operation efficiency of the system.

[0051] 2、Intelligent path planning and operation optimization: The present application can automatically generate the optimal path and operation strategy by combining environmental state modeling and policy optimization mechanism. Unlike the fixed path planning in existing methods, the present application introduces a reinforcement learning model in real-time path planning, which not only avoids obstacles, but also dynamically selects the optimal operation path, reducing the empty time and repeated operation of the robot arm. The introduction of this module makes the system have a shorter path and more concise action during the reagent bottle taking and placing process, thereby significantly improving the operation efficiency and safety of the robot arm.

[0052] 3、Reagent state tracking and layout optimization: The present application can dynamically record the information of all reagent bottles, such as storage location, usage frequency and expiration date, guide intelligent layout optimization, and automatically adjust the placement position of reagent bottles in the reagent rack. Compared with the traditional reagent management method relying on manual or semi-automatic equipment, the present application can help to intelligently adjust the layout position of reagent bottles, preferentially place high-usage-frequency reagents in the area convenient for taking and placing, and remind and handle the reagent bottles about to expire, thereby improving the intelligent level of overall management. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The flowchart of the laboratory reagent bottle position dynamic adjustment and path planning method in an embodiment of the present application.

[0054] Figure 2 The architecture diagram of the deep Q network model in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described below in conjunction with the drawings and specific embodiments, but not as a limitation of the present application.

[0056] In the following detailed description, many specific details are set forth in order to provide a more thorough understanding of the present application. However, it will be apparent to one skilled in the art that the well-known algorithms do not show detailed processes in order to avoid obscuring the main idea of the present application.

[0057] In addition, the order of actions, steps, etc. in the devices and methods shown in the claims, specification and drawings can be implemented in any order as long as there is no specific order limitation and the output of the previous process is not used in the subsequent process.

[0058] Embodiment 1

[0059] Reference Figure 1 The present embodiment provides a laboratory reagent bottle position dynamic adjustment and path planning method, comprising:

[0060] Establishing a lab environment state model: the environment state model includes but is not limited to reagent racks, reagent bottle operating mechanical arms, real-time positions of reagent bottles, and operating trajectories of the mechanical arms.

[0061] Specifically, the environment state model includes the following elements:

[0062] State Space:

[0063] State Space represents all possible states of the mechanical arm in the lab environment. Each state is described by the following features:

[0064] Layout information of reagent racks: represents the positions, types, and placement states of various reagent bottles in the reagent racks.

[0065] Position and pose of the mechanical arm: describes the current spatial position (such as coordinates) of the mechanical arm and the rotation angle of its gripper.

[0066] Position and attributes of the target reagent bottle: represents the target position of the reagent bottle currently needed to be taken or placed by the mechanical arm and related features (such as label information, weight, etc.).

[0067] Action Space:

[0068] Action Space describes all operations that can be executed by the mechanical arm in each state . Specifically, it includes:

[0069] Moving direction of the mechanical arm (such as up and down, left and right, forward and backward, etc.).

[0070] Gripping action of the mechanical arm (such as grasping, releasing, adjusting force, etc.).

[0071] Path adjustment of the mechanical arm (such as translation, rotation, scaling, etc.).

[0072] Reward Function:

[0073] Reward Function is used to measure the performance of the mechanical arm after executing a certain action . The specific reward strategy is as follows:

[0074] When successfully taking or placing a reagent bottle, a positive reward (such as +10) is given.

[0075] When a collision or drop occurs during the taking or placing process, a negative reward (such as -20) is given.

[0076] The robot arm is rewarded with a path efficiency reward (e.g. +5) when it reaches the target position along the shortest path.

[0077] Through the above definitions of the state space, action space and reward function, the system can model and evaluate the operation of the robot arm in the laboratory.

[0078] Preferably, the data of the state space is derived from a multi-modal perception module constructed on the robot arm operating the reagent bottle: the multi-modal perception module includes multiple sensors such as an RGB camera, a depth camera, a torque sensor and a temperature and humidity sensor; the RGB camera is used to capture the color, label information and surface morphology features of the reagent bottle and generate two-dimensional image data; the depth camera is used to collect the depth information and relative distance of the reagent bottle in three-dimensional space; the torque sensor is used to monitor the contact force of the robot arm when grabbing and moving the reagent bottle to ensure the safety of the operation process; the temperature and humidity sensor is used to monitor the temperature and humidity changes in the laboratory environment in real time to ensure the environmental stability of the reagent during storage.

[0079] The data of the multiple sensors is aligned and synchronized with the time and space coordinates to form complete environmental perception data at the same time point; during data fusion, a method combining Kalman filtering and particle filtering is used to correct the measurement errors of the multiple sensors.

[0080] The multi-modal data fusion method combining Kalman filtering and particle filtering is used in this embodiment to reduce the measurement errors between sensors and effectively improve the accuracy and robustness of environmental perception.

[0081] Kalman filtering is a recursive optimal state estimation method commonly used in linear systems, which is suitable for processing linear sensor data with Gaussian noise. In this embodiment, Kalman filtering is used to fuse the data of the torque sensor and the temperature and humidity sensor to eliminate measurement noise and improve data accuracy. Specifically, Kalman filtering consists of two stages: prediction stage and update stage.

[0082] Prediction stage: according to the state at the last time and the current control input, the state at the current time and the state covariance matrix are predicted:

[0083]

[0084]

[0085] wherein represents the predicted state at time . is the state transition matrix, representing the state transition relationship of the system from the last time to the current time . B is the control matrix, representing the control input Impact on system state. Covariance matrix for predicted state. Process noise covariance matrix.

[0086] Update phase: when new sensor measurements arrive, update the state based on the predicted state and observations:

[0087]

[0088]

[0089]

[0090] where: is the Kalman gain, representing the weight distribution between sensor measurements and predicted states, is the observation matrix, representing the mapping relationship from the state space to the observation space. is the measurement noise covariance matrix. is the updated optimal state estimate. Kalman filtering can effectively reduce noise and correct prediction for linear sensor data, improving the measurement accuracy of torque sensors and temperature and humidity sensors.

[0091] Particle filtering is a nonlinear state estimation technique based on the Monte Carlo method, suitable for processing nonlinear, non-Gaussian distributed complex data. This embodiment uses particle filtering to fuse the visual data of RGB cameras and depth cameras, and processes three-dimensional position information in complex environments.

[0092] Particle filtering represents the probability distribution of the state through a set of discrete particles and their weights, and resamples each particle according to its weight. The particle weight update step is as follows:

[0093] Particle state prediction: let the particle state at the last time be , and the predicted particle state at the current time be:

[0094]

[0095] where f is the state transition function, is the process noise.

[0096] Particle weight update: according to the current observation value and the predicted value of the particle state , update the weight of the particle:

[0097]

[0098] where The likelihood probability between the particle state and the observation value can be estimated by a Gaussian distribution.

[0099] Particle resampling: Resample the particle set according to the weight distribution of the particles, retain the particles with higher weights, and eliminate the particles with lower weights to avoid particle degradation.

[0100]

[0101] wherein is the particle weight, is the state of the particle.

[0102] By fusing visual data through particle filtering, high-precision estimation of reagent bottle positions can be maintained in complex environments such as reagent bottle position changes, occlusions, and stacking, thereby improving the robustness and recognition ability of the system in complex environments.

[0103] Kalman filtering and particle filtering are jointly applied in the present application to form an efficient and accurate multi-modal data fusion method. The specific steps are as follows:

[0104] Linear data processing: Use Kalman filtering to denoise and predict update linear sensor data (such as torque sensors and temperature and humidity sensors), improving the accuracy and stability of the measurement data.

[0105] Nonlinear data processing: Use particle filtering to predict and estimate the state of nonlinear sensor data (such as RGB cameras and depth cameras), and update the weight of the particles according to the current observation value to ensure the fusion accuracy of visual information and three-dimensional depth information.

[0106] Data fusion and joint estimation: Jointly fuse the linear data processed by Kalman filtering and the nonlinear data processed by particle filtering. By weighted averaging, the information from different sensors is integrated into the state estimation at the same time, thereby forming the complete state information of the reagent bottle. Assuming that the linear state estimation is , the nonlinear state estimation is , then the final state estimation is:

[0107]

[0108] wherein, is the weight factor of linear and nonlinear data fusion, which is dynamically adjusted according to the complexity of the laboratory environment and the noise level of the sensor to ensure that the fused state estimation can maintain high robustness in various complex environments.

[0109] Adaptive dynamic adjustment mechanism: the embodiment preferably designs an adaptive dynamic adjustment mechanism, which dynamically adjusts the fusion proportion of linear filtering and nonlinear filtering according to the measurement accuracy and reliability of each sensor in different scenes. For example, when the environmental light condition is good and the label is not blocked, the system will increase the weight of the visual recognition data; when the light condition is poor or the reagent bottle is blocked, the weight of the torque and depth information is increased, so as to ensure the accurate perception of the reagent bottle state.

[0110] Data anomaly detection and correction: in the data fusion process, the embodiment preferably introduces a data anomaly detection and correction mechanism. The specific method is to analyze the residual error of the measurement value output by each sensor and the state estimation after fusion. If the measurement residual error of a sensor exceeds the set threshold, it is considered that the output of the sensor is abnormal. At this time, the system will reduce the weight of the sensor data until its output returns to normal. This mechanism can effectively prevent the overall perception failure caused by the abnormal data of a single sensor.

[0111] In addition, a deep learning visual recognition module based on convolutional neural network and image segmentation technology can be used to process the pictures taken by the camera, identify the type of reagent bottle and extract the label information.

[0112] The multi-modal perception module, visual recognition module and data fusion thereof can realize high-precision identification, positioning and environment state perception of the reagent bottle in a complex laboratory environment, and provide reliable data support for reagent management and operation.

[0113] On the basis of establishing the laboratory environment state model, the embodiment constructs and trains a deep Q network (DQN) model, and then uses the trained deep Q network model for path planning and dynamic optimization of the robot arm.

[0114] The deep Q network model is a deep reinforcement learning algorithm based on Q learning, which approximates the state-action value (Q value function) by introducing a deep neural network to perform policy learning in a high-dimensional state space.

[0115] The input of the deep Q network model is the current state of the robot arm, and the output is the Q value of all possible actions. The constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, uses the shortest path and / or the least number of operations as the target, and updates the Q value using the strategy of separating the target network and the main network. The parameters of the target network are updated from the main network once every fixed number of steps. In the training process of the deep Q network model, the picking and placing operations of the robot arm are simulated, and the strategy is updated according to the operation results.

[0116] More specifically, the basic principle of DQN is:

[0117] DQN approximates the Q-value function by a deep neural network (usually a multi-layer convolutional network or a fully connected network). The Q-value function represents the expected cumulative reward that can be obtained by selecting an action in state . The update formula of the Q-value function is as follows:

[0118]

[0119] where: is the learning rate, which controls the magnitude of the update of the model parameters. is the discount factor, which measures the degree of influence of future rewards on current decisions, and its value range is [0, 1]. is the immediate reward of the current policy, which measures the performance of the current action in state . is the maximum Q-value of all actions in the next state , which represents the maximum return that can be obtained by selecting the optimal action in the next state.

[0120] As shown in Figure 2 , the network structure of DQN consists of the following key components:

[0121] In this invention, the input of the DQN network is the current state of the robot arm, and the output is the Q-value of all possible actions. env specifically refers to the laboratory environment state model composed of state space and action space information. The specific network structure is as follows:

[0122] The input layer is the feature vector of the current state , which contains the position of the robot arm, the target position of the reagent bottle, and the relevant information of the laboratory environment.

[0123] The hidden layer network uses a three-layer fully connected network (Fully Connected Network), each layer contains 128 neurons, and uses ReLU activation function for non-linear mapping:

[0124]

[0125] The number of neurons in the output layer is equal to the number of executable actions in the action space , and each neuron corresponds to the Q-value of an action. Through network forward propagation and Q-value calculation, the model can generate the Q-value of each action according to the current state , so as to select the optimal action.

[0126] Q-value update and experience replay:

[0127] In order to improve the stability and efficiency of model training, this paper introduces an experience replay mechanism. The specific approach is as follows:

[0128] Establishment of experience replay pool:

[0129] During training, the model converts the quadruple of each interaction into The experience replay pool is stored in the experience replay pool. The size of the experience replay pool is set to 10,000, which can store the most recent 10,000 interaction experiences.

[0130] Randomly sample training data:

[0131] During each training session, a small batch of data (usually set to 32) is randomly sampled from the experience replay pool to update the Q value. This breaks the temporal correlation between the training data and improves the generalization ability of the model.

[0132] Separation of the target network from the main network:

[0133] DQN uses target network and the main network The parameters of the target network are updated from the main network every fixed number of steps (preferably 1000 times) to avoid frequent oscillations of the Q-value function during training.

[0134] Training and strategy updates:

[0135] Initialize the Q network parameters:

[0136] Randomly initialize the main network Parameters , and copy a copy to the target network .

[0137] Interaction and experience storage:

[0138] Let the robot arm interact in the laboratory environment and choose actions according to the ε-greedy policy: The probability of randomly selecting an action (exploration) is The probability of selecting the action corresponding to the maximum Q value (utilization). As the training progresses, it gradually decreases The value of (e.g., from 1 to 0.1) can be reduced to increase the probability of using the optimal strategy.

[0139] Q value update and training:

[0140] A small batch of data is randomly sampled from the experience replay pool, the target Q value of each sample is calculated, and the following loss function is minimized:

[0141]

[0142] Where the target Q value Is calculated as follows:

[0143]

[0144] And Represents the parameters of the main network and the target network, respectively.

[0145] Policy update and target network update:

[0146] Every fixed number of steps, the parameters of the main network Are copied to the target network And the current policy is updated.

[0147] Through the above steps, the DQN model can adaptively generate the optimal operation policy through continuous interaction and learning in a complex environment, achieving dynamic path planning and pick-and-place optimization of the robotic arm.

[0148] Training of the DQN network:

[0149] In order to verify the effectiveness of the dynamic path planning module based on multi-modal perception, deep learning visual recognition and deep reinforcement learning in the present application, we carried out detailed model training and experimental testing. The following is the specific training process and experimental settings.

[0150] Model training parameters:

[0151] Learning rate (Learning Rate, α): set to 0.001, controls the learning speed of the model, ensuring that the model can gradually converge during gradient descent.

[0152] Discount factor (Discount Factor, γ): set to 0.95, representing the weight proportion of future rewards. This parameter is used to measure the influence of long-term rewards on current decisions, so that the model pays more attention to long-term benefits when planning paths.

[0153] Exploration rate (Exploration Rate, ε): initial value is 1, gradually linearly decays to 0.1. The ε-greedy strategy is used to randomly select actions with a probability of ε to explore new paths, and to select optimal actions with a probability of 1−ε to gradually shift to utilizing the learned strategy.

[0154] Experience replay pool capacity: set to 10,000, storing the most recent 10,000 interaction data to ensure sample diversity and break the temporal correlation between data.

[0155] Batch Size: Set to 32. 32 samples are randomly sampled from the experience replay pool each time for small batch training.

[0156] Target network update frequency: set to 1000 steps, copy the parameters of the main network to the target network to improve the stability of Q value update.

[0157] Training environment:

[0158] Training was performed on an NVIDIA GeForce RTX 4060 GPU for faster training, equipped with an Intel i7-14700HX CPU and 32GB of memory. Python was used as the programming language, and the PyTorch deep learning framework was used. Parallel computing accelerated the model training process on large datasets, enabling it to complete training in a reasonable time.

[0159] Data integration and training process:

[0160] In order to utilize the data of the multimodal perception module and the deep learning visual recognition module, the following data processing and integration methods are used during training:

[0161] State input: The RGB image, depth information, torque data, and temperature and humidity data collected by the multimodal perception module are preprocessed and input into the DQN model as a representation of the robot's environmental state. This data helps the model better perceive the laboratory environment and the specific location of the reagent bottles.

[0162] Visual data fusion: The reagent bottle label information (such as color, shape, etc.) output by the deep learning visual recognition module is used as additional state features to help DQN more accurately determine the type of target reagent bottle and its optimal operation strategy.

[0163] Interaction and experience playback: The state of each robot arm after operation (including success rate, path length, operation time and other information) is recorded as a four-tuple The samples saved in the experience replay pool are randomly sampled and used for subsequent training to improve data utilization.

[0164] DQN training is achieved through the following steps:

[0165] Initialize Q network parameters: Randomly initialize the parameters of the main network Q and copy them to the target network to ensure the stability of the initial Q value.

[0166] Exploration and exploitation strategy: In the early stage of training, a high ε value (ε = 1) is used to conduct extensive exploration to accumulate diverse operation data; as the training progresses, the ε value is gradually decayed, and the current strategy is gradually shifted to exploitation.

[0167] Loss function optimization: Randomly sample from the experience replay pool, calculate the target Q value of each sample, and update the network weights by minimizing the following loss function:

[0168]

[0169] where the target Q value , and represent the parameters of the main network and the target network, respectively.

[0170] Target network update: Every 1,000 steps, the parameters of the main network are copied to the target network to enhance the stability of the training process.

[0171] After the DQN model is trained, we verify the model through different indicators:

[0172] Path optimization effect: The path generated by the DQN model after training reduces the path length by about 30% compared to the random path, indicating that the model can more intelligently select the optimal path.

[0173] Operation success rate: After testing on multiple simulated reagent rack layouts, the model's pick-and-place task success rate reached 96%, indicating that under most experimental conditions, the model can accurately complete the reagent bottle operation.

[0174] Efficiency improvement: The average operation time of the model in completing the pick-and-place task is shortened by 25%, indicating that after training with multi-modal perception and visual recognition data, the DQN model has a more comprehensive understanding of the laboratory environment, thereby effectively improving the operation efficiency.

[0175] Through the dynamic update of the laboratory environment state model, the DQN model can continuously optimize and improve during training, making the path planning of the robotic arm in a complex laboratory environment more intelligent and accurate, thereby improving the overall operation efficiency and reliability of the system. These training results demonstrate the potential of the invention in practical applications, providing an advanced solution for intelligent laboratory management.

[0176] Example 2

[0177] The embodiment provides a laboratory reagent bottle position dynamic adjustment and path planning method system for realizing the laboratory reagent bottle position dynamic adjustment and path planning method as described in Example 1, comprising:

[0178] A laboratory environment state model, including but not limited to reagent racks, reagent bottle operating mechanical arms, real-time positions of reagent bottles, and operating trajectories of the mechanical arms;

[0179] A deep Q network model and a training module thereof, a deep reinforcement learning algorithm based on Q learning, which approximates a state-action value (Q value function) by introducing a deep neural network, performs path planning and dynamic optimization of a mechanical arm:

[0180] The input of the deep Q network model is the current state of the mechanical arm , and the output is the Q value of all possible actions; the deep Q network model is configured to set a reward mechanism, introduce an experience replay mechanism, use the shortest path and / or the least number of operations as the target, and update the Q value using a strategy of separating the target network and the main network, the parameters of the target network being updated from the main network every fixed number of steps; in the training process of the deep Q network model, the picking and placing operations of the mechanical arm are simulated, and the strategy is updated according to the operation results.

[0181] As a preferred technical solution, the laboratory environment state model also tracks and records information including the usage frequency, expiration date, storage location, and environmental conditions of reagent bottles in real time, and uploads them to a laboratory management central controller. The central controller generates management instructions (such as mechanical arm picking and placing path planning, storage environment adjustment, alarm prompts, etc.) based on the state information, and sends them to the corresponding execution modules. At the same time, the central controller can combine the optimal path results of the path planning system to globally optimize the reagent rack layout, ensuring that the mechanical arm has the shortest path and the least operation time when performing picking and placing tasks.

[0182] The above-mentioned laboratory reagent bottle position dynamic adjustment and path planning method can be embodied in the form of a computer program product or a software functional unit. If the above-mentioned laboratory reagent bottle position dynamic adjustment and path planning method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Therefore, the technical solution essentially or the part that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing an electronic system (which can be a personal computer, a server, or a network system, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0183] Those skilled in the art can understand that the units, i.e. algorithm steps, of each example described in combination with the present embodiment can be realized in electronic hardware or in combination of computer software and electronic hardware. Whether the functions are realized in hardware or software mode depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0184] Those skilled in the art should understand that the skilled person can realize variations in combination with the prior art and the above-described embodiments, which are not described herein. Such variations do not affect the essential content of the present application and are not described herein.

[0185] The preferred embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and that systems and structures not described in detail should be understood to be implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the technical solutions of the present application, or modify them as equivalent embodiments of equivalent changes, without departing from the scope of the technical solutions of the present application, using the methods and technical contents disclosed above, which do not affect the essential content of the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiments in accordance with the technical essence of the present application, without departing from the content of the technical solutions of the present application, still belongs to the scope of protection of the technical solutions of the present application.

Claims

1. A method for dynamic position adjustment and path planning of laboratory reagent bottles, characterized in that: include: Establishing a laboratory environment state model: the environment state model includes but is not limited to the reagent rack, the reagent bottle operation robot arm, the real-time position of the reagent bottle and the operation trajectory of the robot arm; Build and train a deep Q-network model, and then use the trained deep Q-network model for path planning and dynamic optimization of the robotic arm; The deep Q-network model is a deep reinforcement learning algorithm based on Q-learning. It introduces a deep neural network to approximate the state-action value, i.e., the Q-value function, and performs strategy learning in a high-dimensional state space. The input of the deep Q network model is the current state of the robotic arm , the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robotic arm is simulated, and the strategy is updated according to the operation results.

2. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 1, characterized in that: The environmental state model includes the following elements: State Space: State Space Represents all possible states of the robot in the laboratory environment; each state Described by the following characteristics: Reagent rack layout information: indicates the position, type and placement status of each reagent bottle in the reagent rack; The position and posture of the robot arm: describes the current spatial position of the robot arm and the rotation angle of its gripper; The position and properties of the target reagent bottle: indicates the target position and related features of the reagent bottle that the robot currently needs to pick up and place; Action Space: Action Space Describe the robot arm in each state All operations that can be performed under , including: The movement direction of the robot arm; the gripping action of the robot arm; the path adjustment of the robot arm; Reward Function: Reward Function Used to measure the robot arm's performance in performing a certain action After the performance, the reward strategy is as follows: When the reagent bottle is successfully taken or placed, positive rewards are given; If a collision or drop occurs during the picking and placing process, a negative reward will be given; When the robot arm reaches the target position along the shortest path, a path efficiency reward is given.

3. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 2, characterized in that: The state space and action space data are derived from a multimodal perception module built on the robotic arm that operates the reagent bottle. The multimodal perception module includes multiple sensors, including an RGB camera, a depth camera, a torque sensor, and a temperature and humidity sensor. The data from these multiple sensors are aligned and synchronized with time and space coordinates to form complete environmental perception data at the same point in time. During the data fusion process, a method combining Kalman filtering and particle filtering is used to correct the measurement errors of the multiple sensors; a deep learning visual recognition module based on convolutional neural networks and image segmentation technology is used to identify the types of reagent bottles and extract label information.

4. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 2, characterized in that: The Q-value function of the deep Q-network model Indicates that the status Next select action The expected cumulative reward that can be obtained; the update formula of the Q-value function is as follows: ， in: is the learning rate, which controls the amplitude of model parameter updates; Discount Factor is used to measure the impact of future rewards on current decisions, with a value range of [0, 1]; is the immediate reward of the current strategy, which measures the current action In state The performance of the following; For the next state The maximum Q value of all actions in the state represents the maximum reward that can be obtained by choosing the best action in the next state.

5. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 4, characterized in that: The deep Q network model includes an input layer, a hidden layer network, and an output layer; the input layer is the current state The hidden layer network adopts a three-layer fully connected network, each layer contains several neurons, and uses the ReLU activation function for nonlinear mapping: ; The number of neurons in the output layer is equal to the action space The number of executable actions in the network, each neuron corresponds to the Q value of an action; through network forward propagation and Q value calculation, the model can Generate each action The Q value of the experience replay is used to select the optimal action; the experience replay mechanism includes the establishment of an experience replay pool, and randomly sampling a training data from the experience replay pool to update the Q value.

6. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 5, characterized in that: The training and strategy update of the deep Q network model includes the following processes: Initialize Q network parameters: Randomly initialize the main network Parameters , and copy a copy to the target network ; Let the robot arm interact in the laboratory environment and choose actions according to the ε-greedy policy: The probability of randomly selecting an action is The probability of selecting the action corresponding to the maximum Q value is gradually reduced as the training progresses. The value of Q value update and training: Randomly sample mini-batches of data from the experience replay pool, calculate the target Q value for each sample, and minimize the following loss function: ， Among them, the target Q value The calculation is as follows: ， and Represent the parameters of the main network and the target network respectively; Policy update and target network update: Every fixed number of steps, the parameters of the main network are adjusted Copy to target network and update the current policy.

7. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 6, characterized in that: The experience replay pool capacity is set to 10,000, storing the most recent 10,000 interaction data.

8. The method for dynamic position adjustment and path planning of laboratory reagent bottles according to claim 6 or 7, characterized in that: The update frequency of the target network is set to 1000 steps, and the parameters of the main network are copied to the target network every 1000 steps.

9. Laboratory reagent bottle position dynamic adjustment and path planning method system, characterized in that: A method for dynamically adjusting the position of a laboratory reagent bottle and planning a path according to any one of claims 1 to 8, comprising: Laboratory environment state model, including but not limited to reagent racks, reagent bottle manipulation robotic arms, the real-time positions of reagent bottles, and the operation trajectory of the robotic arms; The deep Q-network model and its training module, based on the Q-learning deep reinforcement learning algorithm, introduces a deep neural network to approximate the state-action value, i.e., the Q-value function, to perform path planning and dynamic optimization of the robotic arm: The input of the deep Q network model is the current state of the robotic arm , the output is the Q value of all possible actions; the constructed deep Q network model sets a reward mechanism, introduces an experience replay mechanism, adopts the shortest path and / or the least number of operations as the goal, and adopts a strategy of separating the target network and the main network to update the Q value, and the parameters of the target network are updated from the main network once every fixed number of steps; during the training process of the deep Q network model, the pick-and-place operation of the robotic arm is simulated, and the strategy is updated according to the operation results.

10. The laboratory reagent bottle position dynamic adjustment and path planning system according to claim 9, characterized in that: The laboratory environment status model tracks and records information including the frequency of use, expiration date, storage location and environmental conditions of reagent bottles in real time, and uploads it to the laboratory management central controller.

Images