Automated 6-axis manipulator pipette precision dispensing and inspection system with llm-based data processing and real-time coordinate extraction and method thereof

Abstract

An automated system for precision dispensing and inspection of a 6-axis manipulator pipette through LLM-based data processing and real-time coordinate extraction, and a method thereof are disclosed. The automated system for precision dispensing and inspection of a 6-axis manipulator pipette includes a 6-axis robotic arm and a 3D camera. It analyzes solution information through a large language model (LM) to determine the volume of the solution to be dispensed, adjusts the volume of the pipette mounted on the robotic arm to draw up the solution in the set volume, extracts the dispensing position of the solution in real-time using the 3D camera to move the robotic arm to the center of the beaker, and controls the robotic arm according to the center coordinates of the beaker to dispense the solution to a target position.

Description

Technology Field

[0001] The following description concerns a precision solution dispensing automation system using a 6-axis robotic arm and a pipette. Background Technology

[0002] With the rapid advancement of the life sciences and pharmaceutical fields, high-precision and high-throughput liquid sample processing is essential in biotechnology, cell engineering, and genetic engineering research. Research processes in these fields require accurate and consistent liquid handling, and manual pipetting methods have limitations in meeting these advanced demands.

[0003] For example, Korean published patent No. 10-2013-0081954 (published on July 18, 2013) discloses a multi-dispensing method capable of continuously dispensing a solution.

[0004] Conventional manual pipetting methods vary in performance depending on the operator's skill level, and rapid and stable operation is difficult when sample volumes are high. Additionally, there is a problem in that it is difficult to guarantee consistent experimental results due to high fatigue caused by repetitive tasks.

[0005] While the emergence of automated pipetting workstations has partially resolved these issues, the level of automation remains limited due to the continued need for manual intervention during the sample positioning and transfer stages. Consequently, there is a demand for better automation systems to enhance operational efficiency and accuracy in laboratories.

[0006] Robotic liquid handling systems combine automated pipetting and positioning to reduce human intervention while providing high accuracy and repeatability, and are widely utilized in biology, chemistry, and pharmaceutical laboratories. However, existing systems are limited to performing simple, repetitive tasks using complex instruments; precise control is difficult due to a lack of real-time error detection and position verification for various situations that occur during experiments. Furthermore, the systems occupy a large physical space or are bulky, requiring significant space for installation and operation, which presents spatial limitations for application in small to medium-sized laboratory environments.

[0007] Conventional automated pipetting systems offer limited functionality, performing operations only at fixed locations, making it difficult to automate the various variables and operations required for complex chemical experiments. In particular, existing systems are prone to issues regarding experimental reliability and completeness due to insufficient capabilities to detect situational changes or errors during the experiment, as well as inadequate verification of the pipetting process. Consequently, there is a growing need for intelligent automation systems that include advanced error detection and location verification.

[0008] With the recent increase in the need for data-driven customized solutions, the importance of systems that integrate and analyze diverse data to provide results optimized for users is being emphasized. However, existing automated analysis systems generally operate by deriving results based on predefined criteria or analyzing data using fixed algorithms. Because these systems perform analysis based on limited variables, they have limitations in that they cannot adequately reflect the unique characteristics of users.

[0009] Personalized recommendations require the ability to integrate and comprehensively analyze various data, such as images and survey responses; however, existing algorithms process this data individually or analyze it in a standardized manner, making it difficult to propose optimal solutions that consider the correlations between data. The problem to be solved

[0010] Through an application utilizing a large language model (LLM), a system can be provided that automatically classifies recommended solutions based on the user's skin condition by transmitting them to a robotic arm, and dispenses the solution using a pipette after reviewing the dispensing process.

[0011] A precision solution dispensing automation system using a 6-axis robotic arm and a pipette can be designed to dispense precisely by automating volume control through a stepping motor during the solution dispensing process. means of solving the problem

[0012] A solution dispensing automation system implemented by a computer comprises a 6-axis robotic arm and a 3D camera, and at least one processor implemented to execute commands readable by a computer device, wherein the at least one processor analyzes solution information through a large language model (LM) and determines the volume of the solution to be dispensed, adjusts the volume of a pipette mounted on the robotic arm to draw up a set volume of the solution, extracts the dispensing position of the solution in real time using the 3D camera and moves the robotic arm to the center of the beaker, and controls the robotic arm according to the center coordinates of the beaker to dispense the solution at a target position.

[0013] According to one aspect, the at least one processor can derive a recommended solution by analyzing the user's skin condition through the LLM.

[0014] According to another aspect, the at least one processor can derive a recommended solution by analyzing the user's skin condition through the LLM, which is fine-tuned based on text data and image data.

[0015] According to another aspect, the at least one processor analyzes solution information through a multi-agent composed of a first LLM and a second LLM, analyzes a user's skin image through the first LLM to convert the skin condition into text form, and analyzes the text result from the first LLM and the user's survey data through the second LLM to derive recommended ingredients.

[0016] According to another aspect, the at least one processor uses vision technology to examine whether there is a tip at a target position before the robot arm assists the tip, and if a tip is present at the target position, attaches the tip to the robot arm after fine-tuning it to the new tip's origin value by comparing it with an initially set tip origin value, and if no tip is present at the target position, moves to a tip located next to the target position.

[0017] According to another aspect, the at least one processor can detect the beaker and the solution in each beaker using the 3D camera and store the position values, and use the 2D camera mounted on the robot arm to locate the target beaker and fine-tune the robot arm.

[0018] According to another aspect, the at least one processor controls the volume of a solution to be dispensed using a gripper including a stepping motor and a pipette, the gripper drives the stepping motor to control the volume of the pipette, and through the combination of the gripper and the stepping motor, the solution dispensing operation and the object grasping operation can be performed in one device.

[0019] According to another aspect, the stepping motor adjusts the capacity of the pipette to a set value, and the gripper can be designed with a 2-finger structure for solution dispensing and object gripping operations.

[0020] According to another aspect, it may include a structure for replacing the gripper to enable pipette replacement for using pipettes of different capacities.

[0021] According to another aspect, the at least one processor can detect text on a label attached to a solution beaker through the 3D camera and an optical character recognition (OCR) technique, calculate the center coordinates of the detected text box to extract the 3D position of the center coordinates, and provide 3D position information for the work target point of the robot arm based on the 3D position of the center coordinates.

[0022] According to another aspect, the at least one processor can perform coordinate mapping by aligning the depth frame and RGB frame of the 3D camera, and filter the depth data of the 3D camera by applying spatial filtering and temporal filtering techniques.

[0023] According to another aspect, the at least one processor may calculate the three-dimensional position of pixel coordinates using internal coefficients of the 3D camera and convert the three-dimensional position of pixel coordinates into robot coordinates and provide them to the robot arm.

[0024] According to another aspect, the at least one processor can detect a pipette tip through an object detection model that supports instance segmentation, calculate the edges of the pipette tip through a contour detection technique, calculate the inclination and principal axis of the pipette tip through PCA (principal component analysis), define a coordinate system based on the inclination and principal axis of the pipette tip to distinguish the top and bottom of the pipette tip by comparing the width of the x-axis based on the same y-axis, calculate gripping coordinates based on the top and bottom information of the pipette tip, and place the pipette tip in a rack based on the corresponding coordinates.

[0025] A solution dispensing automation method performed in a solution dispensing automation system comprising a 6-axis robotic arm and a 3D camera comprises: a step of analyzing solution information through a large language model (LM) and determining the volume of the solution to be dispensed; a step of aspirating a set volume of the solution by adjusting the volume of a pipette mounted on the robotic arm; a step of moving the robotic arm to the center of a beaker by extracting the dispensing position of the solution in real time using the 3D camera; and a step of dispensing the solution at a target position by controlling the robotic arm according to the center coordinates of the beaker.

[0026] A computer program stored on a computer-readable recording medium for executing a solution dispensing automation method on a computer device, wherein the solution dispensing automation method is performed in a solution dispensing automation system comprising a 6-axis robotic arm and a 3D camera, and comprises the steps of: analyzing solution information through a large language model (LM) and determining the volume of the solution to be dispensed; adjusting the volume of a pipette mounted on the robotic arm to draw up a set volume of the solution; using the 3D camera to extract the dispensing position of the solution in real time and moving the robotic arm to the center of the beaker; and controlling the robotic arm according to the center coordinates of the beaker to dispense the solution to a target position. Effects of the invention

[0027] According to embodiments of the present invention, the flexibility of the work can be increased by using a 3D camera to extract coordinates that change in real time rather than fixed coordinates.

[0028] According to embodiments of the present invention, automation errors can be minimized by adding review processes such as tip detection and coordinate correction.

[0029] According to embodiments of the present invention, the completeness of the automation operation can be improved by controlling the pipette volume using a stepping motor.

[0030] According to embodiments of the present invention, the processing capacity of a dispensing system can be expanded by using pipettes of various capacities through gripper replacement.

[0031] According to embodiments of the present invention, manual work can be reduced by implementing a process of automatically filling a pipette tip rack through pipette tip recognition technology. Brief explanation of the drawing

[0032] FIG. 1 is a block diagram illustrating an example of the internal configuration of a computer device in an embodiment of the present invention. FIGS. 2 and 3 illustrate an example of a precision solution dispensing automation system using a 6-axis robotic arm and a pipette in an embodiment of the invention. FIG. 4 illustrates a precision solution dispensing automation process using a 6-axis robotic arm and a pipette in an embodiment of the present invention. FIG. 5 illustrates the process of data analysis and extraction of blended solution components using LLM in one embodiment of the present invention. FIG. 6 illustrates an example of a pipette tip detection process using a 2D camera in an embodiment of the present invention. FIG. 7 illustrates an example of a beaker recognition process in an embodiment of the present invention. FIG. 8 illustrates an example of a pipette tip rack replenishment process in an embodiment of the present invention. FIG. 9 illustrates an example of a gripper structure designed to enable solution dispensing and object gripping operations in one embodiment of the present invention. FIG. 10 illustrates an example of a pipette gripper structure in one embodiment of the present invention. Specific details for implementing the invention

[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0035] Embodiments of the present invention relate to a precision solution dispensing automation system using a 6-axis robotic arm and a pipette. By using an application with LLM to transmit a recommended solution to the robotic arm according to the user's skin condition for automatic classification and dispensing the solution with a pipette after reviewing the solution dispensing process, precise dispensing can be achieved by automating volume control through a stepping motor during the solution dispensing process.

[0036] The present invention is intended to minimize human error that may repeatedly occur in liquid dispensing operations and to resolve the uncertainty of results resulting from technological changes.

[0037] Furthermore, the present invention aims to minimize errors caused by operator experience or sensitivity by providing consistent angles and depths in handling small amounts of solutions. This consistency enhances the reproducibility of experimental results and supports stable pipetting.

[0038] In addition, the present invention minimizes the physical radius of the robot through the design of an efficient movement path of the robot arm, thereby enabling the efficient use of a solution dispensing automation system even in small and medium-sized laboratories with limited space.

[0039] Furthermore, the present invention performs calculation and review processes to accurately detect the position of the solution and pipette tip and to move to target coordinates through a camera-based real-time robotic arm control system. As a result, the robot can verify the precise location of the target object in real time and aims to provide a system that enables automated solution dispensing operations.

[0040] In addition, the present invention is designed to automatically detect abnormal situations that may occur during the solution dispensing process, and checks whether the pipette tip is correctly mounted in the tip rack through a 2D camera review system, detects the circular shape and center coordinates of the pipette tip, and compares them with the target coordinates of the robot arm.

[0041] In addition, the present invention proceeds with the step of recognizing the edge of the beaker to extract the center coordinates of the container. Through this, the automated system can perform a self-verification function without human intervention, thereby maximizing the accuracy and reliability of the operation.

[0042] In addition, the present invention aims to improve the precision and efficiency of experimental operations by integrating a gripper capable of performing pipetting operations and a 2-finger capable of performing object gripping operations into a single system.

[0043] Furthermore, the present invention aims to efficiently perform volume control tasks required in experimental environments by enabling the flexible use of pipettes of different capacities through gripper replacement. Through this, the invention seeks to expand the processing range of dispensing operations by utilizing pipettes of the necessary capacity to suit experimental conditions.

[0044] In addition, the present invention enables precise dispensing of small amounts of solution by controlling the volume of the pipette using a stepping motor for high-precision pipetting, thereby enabling the automation of solution control and dispensing operations required in various pipetting experiments.

[0045] Furthermore, the present invention aims to improve the accuracy of analysis and reduce inaccurate responses and AI hallucinations by applying LLM fine-tuning and a multi-agent system, thereby deriving optimal recommendation results that reflect the unique characteristics of the user.

[0046] A solution dispensing automation system according to embodiments of the present invention may be implemented by at least one computer device, and a solution dispensing automation method according to embodiments of the present invention may be performed through at least one computer device included in the solution dispensing automation system. At this time, a computer program according to one embodiment of the present invention may be installed and run on the computer device, and the computer device may perform a solution dispensing automation method according to embodiments of the present invention under the control of the run computer program. The above-described computer program may be stored on a computer-readable recording medium to be combined with the computer device to execute the solution dispensing automation method on the computer.

[0047] FIG. 1 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, a solution dispensing automation system according to embodiments of the present invention can be implemented by a computer device (100) illustrated in FIG. 1.

[0048] As illustrated in FIG. 1, the computer device (100) may include a memory (110), a processor (120), a communication interface (130), and an input / output interface (140) as components for executing a solution dispensing automation method according to embodiments of the present invention.

[0049] Memory (110) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device (100) as a separate permanent storage device distinct from memory (110). Additionally, an operating system and at least one program code may be stored in memory (110). These software components may be loaded into memory (110) from a computer-readable recording medium separate from memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, software components may be loaded into memory (110) through a communication interface (130) rather than a computer-readable recording medium. For example, software components can be loaded into the memory (110) of the computer device (100) based on a computer program installed by files received through the network (160).

[0050] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via memory (110) or a communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a recording device such as memory (110).

[0051] The communication interface (130) may provide a function for the computer device (100) to communicate with other devices through a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer device (100) according to program code stored in a recording device such as memory (110) may be transmitted to other devices through the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (100) through the communication interface (130) of the computer device (100) via the network (160). Signals, commands, data, etc. received through the communication interface (130) may be transmitted to the processor (120) or memory (110), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (100) may further include.

[0052] The communication method is not limited and may include not only communication methods utilizing communication networks (e.g., mobile communication networks, wired internet, wireless internet, broadcasting networks) that the network (160) may include, but also short-range wired / wireless communication between devices. For example, the network (160) may include any one or more networks such as a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Additionally, the network (160) may include any one or more network topologies such as a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, but is not limited thereto.

[0053] The input / output interface (140) may be a means for interfacing with an input / output device (150). For example, the input device may include a device such as a microphone, keyboard, camera, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (140) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device (150) may be composed of a computer device (100) and a single device.

[0054] Additionally, in other embodiments, the computer device (100) may include fewer or more components than the components of FIG. 1. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (100) may be implemented to include at least some of the input / output devices (150) described above, or may include other components such as a transceiver, a camera, various sensors, a database, etc.

[0055] Below, specific examples of automated technology for precision dispensing and inspection of 6-axis manipulator pipettes through LLM-based data processing and real-time coordinate extraction will be described.

[0056] The solution dispensing automation system according to the present invention can perform data analysis using an LLM, extract the components of the mixed solution, and execute solution dispensing operations using a 6-axis manipulator. In addition, the solution dispensing automation system according to the present invention can provide a pipetting gripper that controls the volume with a stepping motor for precise solution dispensing operations and a tool changer function for dispensing a wide range of solutions, and can provide the integration of a 2-finger gripper and a pipette gripper to improve work efficiency. In this embodiment, work efficiency can be improved by reviewing using a 3D camera and a 2D camera and by automated rack replenishment to enhance the accuracy and degrees of freedom of the automation system.

[0057] In other words, the present invention can minimize human error and achieve experimental accuracy and reproducibility by implementing an automated solution dispensing system to provide consistent results regardless of the proficiency of pipette operation during the solution dispensing process.

[0058] The solution dispensing automation system according to the present invention may include the following core technologies.

[0059] 1) Accuracy can be improved by integrating image and text analysis using LLM (e.g., GPT-4o), and solution formulation can be carried out by processing data from various perspectives through a multi-agent system.

[0060] 2) Depth data filtered by a depth camera and text location detected by OCR (optical character recognition) and the actual location of the target calculated as a 3D point based on center coordinates can be transmitted to the robot to send movement commands.

[0061] 3) By using a 2D camera, an object detection model (e.g., YOLOv8), a feature extraction technique (e.g., Hough Transform), and an edge detection technique (e.g., Canny Edge Detection), the center of a pipette tip and various shapes of beakers can be precisely recognized and reflected in the position adjustment of the robot arm.

[0062] 4) It may include a pipetting system that precisely controls pipettes of various capacities through the use of a stepping motor and a tool change function.

[0063] 5) Deep learning-based computer vision technology (e.g., Instance Segmentation) can be utilized to individually recognize the position and tilt of pipette tips and automate rack replenishment operations based on this.

[0064] The computer device (100) according to the present embodiment can provide a solution dispensing automation service to a client by accessing a dedicated application installed on the client or a web / mobile site related to the computer device (100). The computer device (100) may be configured with a computer-implemented solution dispensing automation system. For example, the solution dispensing automation system may be implemented in the form of a program that operates independently, or configured as an in-app of a specific application so that it can operate on said specific application.

[0065] The processor (120) of the computer device (100) may be implemented as a component for performing the following solution dispensing automation method. Depending on the embodiment, the components of the processor (120) may be optionally included in or excluded from the processor (120). Additionally, depending on the embodiment, the components of the processor (120) may be separated or merged to represent the function of the processor (120).

[0066] These processors (120) and components of the processor (120) can control a computer device (100) to perform steps included in the following solution dispensing automation method. For example, the processor (120) and components of the processor (120) may be implemented to execute instructions according to the code of an operating system included in memory (110) and the code of at least one program.

[0067] Here, the components of the processor (120) may be representations of different functions performed by the processor (120) according to instructions provided by program code stored in the computer device (100).

[0068] The processor (120) can read necessary instructions from memory (110) in which instructions related to the control of the computer device (100) are loaded. In this case, the read instructions may include instructions for controlling the processor (120) to execute the steps to be described later.

[0069] The steps included in the solution dispensing automation method described below may be performed in a different order than the illustrated order, and some of the steps may be omitted or additional processes may be included.

[0070] FIGS. 2 and 3 illustrate an example of a precision solution dispensing automation system using a 6-axis robotic arm and a pipette in an embodiment of the invention.

[0071] The solution dispensing automation system according to the present invention may include a 2D camera, a 3D camera (depth camera), and a 6-axis robotic arm. FIG. 3 illustrates an example of an actual environment of a precision solution dispensing automation system using a 6-axis robotic arm and a pipette.

[0072] The present invention can provide a robotic arm automated liquid dispensing environment including a 6-axis robotic arm including a 2D camera, a stepping motor, a gripper, a 3D camera, and LLM-based data and a review process.

[0073] FIG. 4 illustrates the process of automated precision solution dispensing using a 6-axis robotic arm and a pipette in an embodiment of the present invention. FIG. 4 shows a flowchart in which the robotic arm completes the system by communicating with the robot-side server and the PC-side client within the system.

[0074] The solution dispensing automation system according to the present invention may include real-time solution detection and coordinate extraction, automatic pipette volume adjustment through a stepping motor, a gripper replacement function, and an automated process for replenishing the pipette tip rack, and is characterized by improved work efficiency and reduced automation errors through review work.

[0075] The solution dispensing automation system according to the present invention can precisely dispense a solution through the following steps.

[0076] In step (a), solution information can be analyzed through LLM-based data and the volume of solution to be dispensed can be determined.

[0077] (b) In step (b), the pipette tip can be removed or attached using a stepping motor and a gripper, and the volume of the pipette can be adjusted to draw in and eject a set volume of solution.

[0078] (c) In step (c), the coordinates of the dispensing position can be extracted in real time using a 3D camera and coordinate correction can be performed to move the robot arm to the center of the container.

[0079] (d) In step, the robot arm can be controlled according to the center coordinates of the container to dispense the solution at the target location.

[0080] In this case, analysis accuracy can be increased by applying fine-tuning and a multi-agent system to GPT-4o, an example of LLM.

[0081] Figure 5 illustrates the process of data analysis and extraction of formulation solution components using LLM, and, for example, a recommended solution can be determined according to the user's skin condition through LLM.

[0082] Errors in analysis results can be minimized by collecting text and image data and fitting large amounts of data using a fine-tuned LLM.

[0083] In this case, by configuring two or more LLMs into a multi-agent system, it is possible to implement a robotic arm automated liquid dispensing system that improves analysis accuracy through data analysis from various perspectives.

[0084] By applying OCR (e.g., Naver Cloud OCR) to a 3D camera, such as a Depth camera (e.g., RealSense D435 camera), it is possible to distinguish solutions and detect coordinates for robot arm control in real time.

[0085] To identify the solution, OCR technology is used to detect the characters on the label attached to the solution beaker in real time, and the center coordinates of the detected text box are calculated and captured with a depth camera to output the 3D position of the corresponding coordinates.

[0086] Through this, precise 3D position information for the robot's work target point is provided based on the detected coordinates, and solution dispensing operations can be performed flexibly in various environments.

[0087] To detect the real-time position of the beaker, the depth frame and RGB frame of the depth camera are aligned to perform coordinate mapping, and then spatial filtering and temporal filtering techniques are applied to improve the accuracy of the depth information.

[0088] Subsequently, the 3D position of the target beaker can be accurately extracted in real time by calculating the 3D position of the pixel coordinates using the internal coefficients of the depth camera and converting it into robot coordinates for output.

[0089] Through this process, the 3D position of the target beaker is accurately extracted in real time, and the design enables flexible solution dispensing operations in various environments.

[0090] The solution can be accurately drawn through automated robotic arm pipetting, which uses a depth camera to detect the beakers freely placed by the user and the solutions in each beaker, stores the position values, moves to the target beaker, detects the edges of the beaker using a 2D camera before drawing the liquid from the beaker, connects the detected edges to find the center point, and fine-tunes the robotic arm again.

[0091] The present invention utilizes vision technology to enable a robot arm to move to attach a tip, and before attaching the tip, check for the presence of the tip at a target location. If the tip is present, it compares it with the origin value of the initially set tip and performs fine adjustments to match the origin value of the new tip to safely attach the tip; if the tip is not present, it moves to a tip located next to it.

[0092] The present invention inspects the condition of pipette tips and solution beakers by adding an inspection system using a 2D camera. In the first step, before the pipette is fitted with a tip, an object detection model such as YOLOv8 is used to detect whether the tip to be fitted is inserted into the tip rack. In the second step, after undergoing grayscale conversion and histogram equalization, the clarity of the pipette tip is enhanced by converting it to a specific color (e.g., red) for a specific threshold value, as shown in FIG. 6. Subsequently, in the final step, the circular shape and center coordinates of the tip are detected using a feature extraction technique such as the Hough Transform, and the position of the robot arm can be precisely adjusted by comparing these with the target coordinates of the robot arm.

[0093] In the beaker inspection process, noise is reduced through preprocessing such as grayscale conversion and Gaussian blurring, and then the edges of the beaker are recognized by performing edge detection techniques (e.g., Canny Edge Detection) and contour detection techniques as shown in Fig. 7. The detected edges are converted to be close to a circle using contour approximation techniques to extract the circle and center coordinates of the beaker. Through this, a multi-purpose detection model is developed that can be applied not only to circular containers but also to containers of various polygonal shapes, thereby enabling accurate recognition of the center of the container even in environments where various containers are used.

[0094] The solution dispensing automation system according to the present invention provides a gripper replacement function for dispensing operations of various volumes and can automate pipette volume control using a stepping motor.

[0095] The present invention designs a pipette-optimized gripper equipped with a stepping motor to enable automatic adjustment of the pipette's capacity. The pipette-optimized gripper integrates the dispensing of solutions and the grasping of objects into a single system through a 2-finger function, thereby enabling high-precision dispensing of fine solutions and handling of various objects. Additionally, pipettes of various capacities can be used through gripper tool changes. A dedicated gripper for grasping pipettes is attached to the parallel gripper used in this system; the upper part of the pipette gripper remains fixed, while the lower part can be designed to be easily attached and detached using magnetism.

[0096] The present invention can automatically adjust the volume using a gripper including a stepping motor and a pipette, and can perform solution dispensing and object gripping operations in an integrated manner in a single device.

[0097] At this time, the gripper drives a stepping motor to precisely control the volume of the pipette, and the combination of the gripper and the stepping motor can be designed to enable precise and stable automation in dispensing small amounts of solution.

[0098] In addition, the stepping motor can accurately adjust the volume of the pipette to a set value, and the gripper is designed with a 2-finger structure to efficiently perform object gripping and solution dispensing operations.

[0099] In this embodiment, a design may be included that allows the gripper to be replaced to use pipettes of various capacities. The solution dispensing automation system according to the present invention has a structure that allows the gripper to be replaced so as to replace pipettes according to the requirements of the working environment. This enables the flexible use of pipettes of different capacities, thereby expanding the scope of experimental work and increasing the processing capacity of dispensing operations.

[0100] The solution dispensing automation system according to the present invention can analyze the user's skin condition using LLM (GPT-4o).

[0101] Referring again to Fig. 5, GPT-4o can be fine-tuned based on image data of the left, right, and front views of the face and approximately 11,000 data points representing the degree of wrinkles, pores, sagging, sensitivity, and pigmentation.

[0102] Data representing skin condition can be quantified and classified into three levels—'Good,' 'Average,' and 'Severe'—using the K-means clustering technique. For each level, the GPT-4o model is utilized to automatically generate sentences describing the corresponding condition. For example, if data regarding 'wrinkles' is classified as 'Severe,' sentences such as "The wrinkles appear very deep and distinct" or "The wrinkles do not smooth out easily or feel fixed" can be generated; 300 sentences are prepared for each level and randomly arranged. Subsequently, the generated sentence data and image data can be combined to fine-tune the GPT-4o model.

[0103] Through the fine-tuning of GPT-4o, inaccurate responses such as "it is difficult to measure skin condition based on images alone" and AI hallucinations can be reduced, enabling the diagnosis of skin condition with high accuracy.

[0104] According to the embodiments, a multi-agent system can be constructed using two GPT-4o models. Two GPT-4o models can be utilized to address the problem where analysis results are biased toward specific data types, such as image or text data, when using a single GPT model. First, a fine-tuned first model analyzes the user's skin image to convert the skin condition into text, and the second model integrates the text results derived by the first model with the user's survey data to derive the final skin condition and recommended ingredients.

[0105] This stepwise analysis method allows for the independent yet complementary utilization of image and text data, reducing the hallucination problem in GPT models and providing more sophisticated and reliable results.

[0106] Finally, the solution dispensing automation system according to the present invention can minimize manual work by recognizing individual pipette tips through instance segmentation and automatically placing them in a tip rack.

[0107] Work efficiency can be improved by automating the replenishment of pipette tip racks. For example, in the process of automating pipette tip rack replenishment, pipette tips can be placed in the rack by performing the following steps.

[0108] Referring to Fig. 8, in step (a), the filling pipette tips can be detected individually through a YOLOv8 model that supports instance splitting.

[0109] In step (b), the edges of the tips can be calculated using a contour detection technique in the order of high-accuracy data among the detected pipette tip data.

[0110] (c) In step, the inclination and major axis of the pipette tip can be calculated through PCA (Principal Component Analysis).

[0111] (d) In step, a coordinate system is defined based on the inclination and major axis of the tip, and the top and bottom of the tip can be distinguished by comparing the width of the x-axis with respect to the same y-axis.

[0112] (e) In step (e), based on the top and bottom information of the tip, the top 1 / 5 point is calculated as the gripping coordinate, and the tip can be placed on the rack based on that coordinate.

[0113] By implementing a function that automatically places pipette tips into a tip rack based on individually detected pipette tip data, reliance on manual work can be minimized, thereby increasing the efficiency and accuracy of pipette tip replenishment and enabling the automation of the entire work process.

[0114] FIG. 9 illustrates an example of a gripper structure designed to enable solution dispensing and object gripping operations in an embodiment of the present invention, and FIG. 10 illustrates an example of a gripper structure for a pipette in an embodiment of the present invention.

[0115] The robotic arm applied to the solution dispensing automation system of the present invention includes a structure that allows pipettes to be easily replaced as needed by adding a magnetic attachment / detachment function.

[0116] The robot arm used in the present invention may include a pipette holder (1), a pipette gripper (2), a stepping motor gripper (3), and a gripper connection part (4). The pipette holder (1) is manufactured for the purpose of hanging a pipette to be replaced, and the pipette gripper (2) may include a structure that allows the pipette to be attached and detached using magnetism. The pipette gripper (2) is designed with a 2-finger structure so that object gripping and solution dispensing operations can be performed efficiently. In addition, the stepping motor gripper (3) refers to a gripper part connected to a stepping motor. Furthermore, the gripper connection part (4) is a member connecting the pipette gripper (2) and the stepping motor gripper (3), and is a part designed to receive the rotational force of the stepping motor to turn the volume control dial of the pipette.

[0117] Therefore, customized formulation ingredients can be extracted through LLM-based user data acquisition and analysis, and a flexible experimental environment can be realized by utilizing depth cameras and OCR to extract the spatial coordinates of each solution and converting them to the robot arm's viewpoint through rigid body transformation. In addition, the accuracy of tip attachment can be improved through tip presence detection and center point output using an object detection model, and errors can be minimized through a review process. Furthermore, errors in the automation process can be minimized through beaker edge detection and center point verification. Moreover, experimental versatility can be ensured with a tool changer that supports multiple micropipettes by integrating a 2-finger gripper and a pipette dispensing gripper and adjusting the volume with a stepping motor, and work efficiency can be maximized through automated rack replenishment.

[0118] The above-mentioned solution dispensing automation system can be utilized in technology fields such as cosmetic manufacturing, PCR machines, and chemical experiments for the automation of precision liquid handling operations and for providing customized solutions optimized for various users.

[0119] The 6-axis robotic arm solution dispensing automation system of the present invention can minimize errors through a step-by-step review process and automatically perform precise solution dispensing operations by increasing operational flexibility through real-time coordinate extraction. This enables the automation of various laboratory tasks requiring precise liquid handling, such as cosmetic manufacturing, PCR machines, and chemical experiments.

[0120] According to the present invention, productivity can be maximized by minimizing manual intervention in pipette operations through an automated system and efficiently reducing working time. In addition, by utilizing an LLM to analyze the skin condition of each user and providing an optimized customized solution, precise dispensing operations tailored to the different needs of each user can be realized.

[0121] According to the present invention, high-accuracy analysis results can be derived with minimal data through fine-tuning of the LLM, and errors in the automation process can be minimized through a review function to increase the reproducibility of experiments and realize advanced precision dispensing operations.

[0122] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0123] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0124] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0125] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0126] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A solution dispensing automation system implemented by a computer, comprising a 6-axis robotic arm and a 3D camera, and at least one processor implemented to execute commands readable by a computer device, wherein the at least one processor analyzes solution information through a large language model (LLM) and determines the volume of the solution to be dispensed, adjusts the volume of a pipette mounted on the robotic arm to draw up a set volume of the solution, extracts the dispensing position of the solution in real time using the 3D camera and moves the robotic arm to the center of the beaker, and controls the robotic arm according to the center coordinates of the beaker to dispense the solution at a target position. Claim 2 A solution dispensing automation system according to claim 1, wherein at least one processor analyzes the user's skin condition through the LLM to derive a recommended solution. Claim 3 A solution dispensing automation system according to claim 1, wherein at least one processor analyzes the user's skin condition through the LLM fine-tuned based on text data and image data to derive a recommended solution. Claim 4 A solution dispensing automation system according to claim 1, wherein the at least one processor analyzes solution information through a multi-agent composed of a first LLM and a second LLM, analyzes a user's skin image through the first LLM to convert the skin condition into text form, and analyzes the text result from the first LLM and the user's survey data through the second LLM to derive recommended ingredients. Claim 5 A solution dispensing automation system according to claim 1, wherein at least one processor examines whether there is a tip at a target position using vision technology before the robot arm assists the tip, and if the tip is at the target position, attaches the tip to the robot arm after fine-tuning it to match the new tip's origin value by comparing it with an initial set tip origin value, and if the tip is not at the target position, moves to a tip located next to the target position. Claim 6 A solution dispensing automation system according to claim 1, wherein at least one processor detects a beaker and the solution in each beaker using the 3D camera and stores a position value, and uses a 2D camera mounted on the robot arm to locate a target beaker and fine-tune the robot arm. Claim 7 A solution dispensing automation system according to claim 1, wherein at least one processor controls the volume of a solution to be dispensed using a gripper including a stepping motor and a pipette, the gripper drives the stepping motor to control the volume of the pipette, and the solution dispensing operation and the object grasping operation are performed in one device through a combination of the gripper and the stepping motor. Claim 8 A solution dispensing automation system according to claim 7, characterized in that the stepping motor adjusts the volume of the pipette to a set value, and the gripper is designed with a 2-finger structure for solution dispensing and object grasping operations. Claim 9 A solution dispensing automation system according to claim 1, characterized by including a structure for replacing a gripper to enable pipette replacement for using pipettes of different capacities. Claim 10 A solution dispensing automation system according to claim 1, wherein the at least one processor detects text of a label attached to a solution beaker through the 3D camera and an OCR (optical character recognition) technique, calculates the center coordinates of the detected text box to extract the 3D position of the center coordinates, and provides 3D position information for the work target point of the robot arm based on the 3D position of the center coordinates. Claim 11 A solution dispensing automation system according to claim 10, wherein at least one processor performs coordinate mapping by aligning the depth frame and RGB frame of the 3D camera, and filters the depth data of the 3D camera by applying spatial filtering and temporal filtering techniques. Claim 12 A solution dispensing automation system according to claim 10, wherein at least one processor calculates a three-dimensional position of pixel coordinates using internal coefficients of the 3D camera, converts the three-dimensional position of pixel coordinates into robot coordinates, and provides them to the robot arm. Claim 13 A solution dispensing automation system according to claim 1, wherein at least one processor detects a pipette tip through an object detection model that supports instance segmentation, calculates the edge of the pipette tip through a contour detection technique, calculates the inclination and principal axis of the pipette tip through PCA (principal component analysis), defines a coordinate system based on the inclination and principal axis of the pipette tip to distinguish the top and bottom of the pipette tip by comparing the width of the x-axis based on the same y-axis, calculates gripping coordinates based on the top and bottom information of the pipette tip, and places the pipette tip in a rack based on the corresponding coordinates. Claim 14 A solution dispensing automation method performed in a solution dispensing automation system comprising a 6-axis robotic arm and a 3D camera, comprising: a step of analyzing solution information through a large language model (LLM) and determining the volume of the solution to be dispensed; a step of aspirating a set volume of the solution by adjusting the volume of a pipette mounted on the robotic arm; a step of moving the robotic arm to the center of a beaker by extracting the dispensing position of the solution in real time using the 3D camera; and a step of dispensing the solution at a target position by controlling the robotic arm according to the center coordinates of the beaker. Claim 15 A computer program stored on a computer-readable recording medium for executing a solution dispensing automation method on a computer device, wherein the solution dispensing automation method is performed in a solution dispensing automation system comprising a 6-axis robotic arm and a 3D camera, and comprises the steps of: analyzing solution information through a large language model (LLM) and determining the volume of the solution to be dispensed; adjusting the volume of a pipette mounted on the robotic arm to draw up a set volume of the solution; using the 3D camera to extract the dispensing position of the solution in real time and moving the robotic arm to the center of the beaker; and controlling the robotic arm according to the center coordinates of the beaker to dispense the solution to a target position.

Images