Robot arm system, and method for driving robot arm system

Abstract

According to some embodiments of the present invention, a robot arm system comprises: a robot arm movement track optimization device, based on 6D path primitive generation using a statistical technique, for generating an optimized pose trajectory; and a painting device, based on pose difference recognition using artificial intelligence (AI), for performing a painting operation by using the optimized pose trajectory as a sample robot arm movement track.

Description

Robot arm system and method for operating the robot arm system

[0001] Cross-citation with related applications

[0002] This application claims the benefit of priority to Korean Patent Application No. 10-2024-0120036, filed September 4, 2024, and Korean Patent Application No. 10-2024-0120044, filed September 4, 2024, the entire contents of which are incorporated herein by reference.

[0003] Technology field

[0004] The present invention is an advanced method for generating a movement track of a robot arm based on motion data collected by a motion tracking method, and relates to a device and method for expressing motion data as a combination of primitives, which are the simplest mathematical form, by utilizing geometric and statistical techniques.

[0005] The present invention relates to a painting automation device and method that can recognize a pose difference between a sample object and a new object using an artificial intelligence (AI) model and use the difference to correct a robot arm movement track.

[0006] Research is underway to automate tasks such as painting the surfaces of large ships and decontaminating nuclear power plant components using robots. To automate these tasks, a robot arm may need to track the 3D position and direction of the nozzle mechanism that sprays the paint or decontamination agent. For example, a method could be utilized where the demonstrator is filmed performing the spraying operation, the position and direction of the sprayer can be extracted, and the robot arm can then track this. However, tracking the demonstrator's exact movements would unnecessarily increase the learning complexity and training time of the robot arm control model, as even minor missteps or hesitations would need to be taken into account.

[0007] Automating the painting process can be accomplished by tracking the movements of the operator or by directly setting the painting path based on the shape of the object. When setting the painting path directly, the accuracy of the path may be reduced if the sample is not positioned correctly or if it shakes, so compensation may be necessary.

[0008] When tracking the motion of a worker, it may be necessary to track the three-dimensional position and three-dimensional spray direction of a nozzle mechanism that sprays paint or decontaminant using a robot arm. Meanwhile, when painting multiple objects of the same shape, even if the three-dimensional position and three-dimensional spray direction of a representative sample object are tracked to generate a representative sample of the robot arm movement track, other objects may be transferred to the painting automation device in different positions and directions than when the sample was formed. Therefore, even if a robot arm movement track that serves as a representative sample is generated, it may be difficult to apply it to all objects.

[0009] One of the objects of the present invention includes optimizing a robot arm movement track to reduce the learning complexity and learning time of a robot arm control model for automating spraying and painting operations using a robot arm.

[0010] One of the objects of the present invention includes providing a device and method capable of correcting a representative sample track by using the pose difference of each object to generate a robot arm movement track that serves as one representative sample and then apply it to all objects.

[0011] According to some embodiments of the present invention, the robot arm system includes a robot arm movement track optimization device based on 6D path primitive generation using a statistical technique, configured to generate an optimized pose trajectory.

[0012] According to some embodiments of the present invention, a method for driving a robot arm system includes the steps of: generating an optimized pose trajectory using a robot arm movement track optimization device based on 6D path primitive generation using a statistical technique; and performing a painting task by treating the optimized pose trajectory as a sample robot arm movement track using a painting device based on pose difference recognition using artificial intelligence (AI).

[0013] According to some embodiments of the present invention, a device for optimizing a robot arm movement track based on generation of 6D path primitives using a statistical technique includes: a photographing unit configured to photograph a state of a task demonstration by a demonstrator during the task demonstration to generate time-series image data; and a first processing unit configured to extract 6D pose values ​​of the task demonstration from the time-series image data to generate time-series pose data, simplify a trajectory for the time-series pose data to generate a robot arm trajectory, and edit duplicate data to generate an optimized pose trajectory.

[0014] According to some embodiments of the present invention, the first processing unit is configured to derive an error range of the time series pose data and perform path smoothing on the time series pose data based on the error range to generate the robot arm pose trajectory.

[0015] According to some embodiments of the present invention, the work demonstration includes a surface painting operation on a 3D object, and the 6D pose values ​​include 3D position values ​​and 3D direction values ​​of the spray painting tool.

[0016] According to some embodiments of the present invention, the first processing unit is configured to perform the path smoothing based on the 3D movement of the spray body sprayed by the spray painting mechanism, the spray amount of the spray body, the work completion rate status of the work demonstration, and the work limit time of the work demonstration.

[0017] According to some embodiments of the present invention, the first processing unit is configured to detect a stationary section of the demonstrator based on the time series pose data, and generate the robot arm pose trajectory by dividing the time series pose data into different paths based on the stationary section.

[0018] According to some embodiments of the present invention, the first processing unit is configured to generate the robot arm pose trajectory as a combination of the stationary section, the straight path section, and the curved path section.

[0019] According to some embodiments of the present invention, the first processing unit is configured to generate the straight path section by combining a first type of non-rotational straight movement, a second type of rotational straight movement, and a third type of non-rotational pure rotation, and to generate the curved path section by combining a fourth type of non-rotational curved movement, a fifth type of rotational curved movement, and a sixth type of rotational spiral movement.

[0020] According to some embodiments of the present invention, the first processing unit is configured to generate segment parameters of the stationary segment, the straight path segment, and the curved path segment, wherein a first segment parameter of the stationary segment includes a stationary time, a second segment parameter of the straight path segment includes a starting point, an ending point, and a progress speed, and a third segment parameter of the curved path segment includes a starting point, an ending point, a progress speed, and a radius of curvature.

[0021] According to some embodiments of the present invention, the first processing unit is configured to generate the optimized pose trajectory by deleting at least some of the stop sections connecting the paths of the robot arm pose trajectory.

[0022] According to some embodiments of the present invention, the first processing unit is configured to classify the stop sections into a first stop section where the demonstrator intentionally stopped and an unnecessary second stop section, and to delete the second stop section to generate the optimized pose trajectory.

[0023] According to some embodiments of the present invention, the first processing unit is configured to calculate a first learning required time of the robot arm control model for the time series pose data before movement track optimization is performed and a second learning required time of the robot arm control model for the optimized pose trajectory after movement track optimization is performed, and to provide a comparison result of the first learning required time and the second learning required time.

[0024] According to some embodiments of the present invention, the first processing unit operates based on a first model that generates a robot arm pose trajectory for automating the task demonstration by performing trajectory simplification on the time series pose data, and a second model that is configured to edit redundant data in the robot arm pose trajectory to generate an optimized pose trajectory.

[0025] According to some embodiments of the present invention, a method for optimizing a robot arm movement track based on generation of 6D path primitives using a statistical technique includes: a step of generating time-series image data by photographing a state of a task demonstration by a demonstrator during the task demonstration by a photographing unit; a step of generating time-series pose data by extracting 6D pose values ​​of the task demonstration from the time-series image data by a first processing unit; a step of generating a robot arm pose trajectory for automating the task demonstration by performing trajectory simplification on the time-series pose data by the first processing unit; and a step of editing redundant data in the robot arm pose trajectory by the first processing unit to generate an optimized pose trajectory.

[0026] According to some embodiments of the present invention, a painting device based on pose difference recognition utilizing artificial intelligence (AI), which is configured to perform a painting work by treating the optimized pose trajectory as a sample robot arm movement track; The painting device based on pose difference recognition utilizing artificial intelligence (AI), comprises: an AI pose extraction unit configured to photograph a 3D object to generate a target image and to extract a 6D pose value of the 3D object from the target image; a second processing unit configured to calculate a pose difference value between a sample pose value prepared in advance based on a sample object and the 6D pose value, and to correct a sample robot arm movement track prepared in advance for the sample pose value based on the pose difference value to a corrected robot arm movement track for the 6D pose value of the 3D object; and a working unit configured to perform a painting work on the 3D object along the corrected robot arm movement track.

[0027] According to some embodiments of the present invention, the painting device further includes a transport unit configured to transport the 3D object so as to sequentially pass through a photographing zone, a buffer zone, and a painting zone.

[0028] According to some embodiments of the present invention, the AI ​​pose extraction unit is configured to generate the target image when the 3D object activates a shooting trigger in the shooting area, and the work unit is configured to perform the painting work when the 3D object activates a work trigger in the work area.

[0029] According to some embodiments of the present invention, the buffer zone is configured to reduce dust generated by the painting work in the painting zone, and the second processing unit is configured to generate the correction robot arm movement track while the 3D object passes through the buffer zone before reaching the painting zone.

[0030] According to some embodiments of the present invention, the second processing unit is configured to generate the corrected robot arm movement track in real time immediately after the pose difference value is calculated by the AI ​​pose extraction unit.

[0031] According to some embodiments of the present invention, the sample object is configured as a digital twin having the same shape as the 3D object, and the second processing unit is configured to compare a sample 6D pose value of the digital twin with the 6D pose value of the 3D object to calculate the pose difference value.

[0032] According to some embodiments of the present invention, the second processing unit is configured to compare a 3D position value of the digital twin with a 3D position value of the 3D object to generate a position difference value, compare a 3D direction value of the digital twin with a 3D direction value of the 3D object to generate a direction difference value, and calculate the pose difference value based on the position difference value and the direction difference value.

[0033] According to some embodiments of the present invention, the 3D object includes a 2D object hanging on a wall, the position difference value includes a height difference value between a height of the 2D object and a height of the digital twin, and the direction difference value includes an angle difference value between an angle of the 2D object and an angle of the digital twin.

[0034] According to some embodiments of the present invention, the second processing unit is configured to generate the corrected robot arm movement track by applying a 3D positional movement and a 3D direction rotation corresponding to the pose difference value to the sample robot arm movement track.

[0035] According to some embodiments of the present invention, the AI ​​correction model is trained to select a first sample 6D pose value that is closest to the 6D pose value of the 3D object among a plurality of sample 6D pose values ​​prepared in advance at intervals of a certain distance and a certain angle, and to set a robot arm movement track corresponding to the first sample 6D pose value as the correction robot arm movement track.

[0036] According to some embodiments of the present invention, a painting method based on pose difference recognition using artificial intelligence (AI) includes: a step of generating a target image by photographing a 3D object through an AI pose extraction unit; a step of extracting a 6D pose value of the 3D object from the target image through the AI ​​pose extraction unit; a step of calculating a pose difference value between a sample pose value prepared in advance based on a sample object and the 6D pose value through a second processing unit; a step of correcting a sample robot arm movement track prepared in advance for the sample pose value based on the pose difference value through the second processing unit into a corrected robot arm movement track for the 6D pose value of the 3D object; and a step of performing a painting operation on the 3D object along the corrected robot arm movement track through a working unit.

[0037] According to embodiments of the present invention, a robot arm movement track can be optimized to reduce the learning complexity and learning time of a robot arm control model for automating spraying and painting operations using a robot arm.

[0038] According to embodiments of the present invention, a device and method can be provided that can correct a representative sample track by using a pose difference of each object to generate a robot arm movement track that serves as one representative sample and then apply it to all objects.

[0039] FIG. 1 illustrates how a robot arm movement track optimization device according to some embodiments operates.

[0040] FIG. 2 illustrates elements constituting a robot arm movement track optimization device according to some embodiments.

[0041] FIG. 3 illustrates a process of generating a movement track of a robot arm by capturing a demonstrator's demonstration motion for a 3D object according to some embodiments.

[0042] FIG. 4 illustrates a three-dimensional position and three-dimensional orientation of a robot arm according to some embodiments.

[0043] FIG. 5 illustrates a process for generating a robot arm 6D primitive trajectory according to some embodiments.

[0044] FIG. 6 illustrates a process for generating a straight path section of a robot arm pose trajectory according to some embodiments.

[0045] FIG. 7 illustrates a process of generating a curved path section of a robot arm pose trajectory according to some embodiments.

[0046] Figure 8 illustrates a process for performing trajectory simplification on time series pose data according to some embodiments.

[0047] FIG. 9 illustrates a process of creating an optimized pose trajectory by editing duplicate data in a robot arm pose trajectory according to some embodiments.

[0048] FIG. 10 illustrates steps of a method for optimizing a robot arm movement track according to some embodiments.

[0049] FIG. 11 illustrates how a pose difference recognition-based painting device operates according to some embodiments.

[0050] FIG. 12 illustrates elements constituting a painting device based on pose difference recognition according to some embodiments.

[0051] FIG. 13 illustrates a process of performing a painting operation on a 3D object according to some embodiments.

[0052] Figure 14 illustrates a process for generating a robot arm movement track according to some embodiments.

[0053] Fig. 15 illustrates a specific form of a robot arm movement track according to some embodiments.

[0054] Figure 16 illustrates a process for calculating a pose difference value for a sample pose value according to some embodiments.

[0055] FIG. 17 illustrates a process of correcting a sample robot arm movement track to a corrected robot arm movement track according to some embodiments.

[0056] Figure 18 illustrates steps of a painting method based on pose difference recognition according to some embodiments.

[0057] FIG. 19 illustrates elements constituting a robot arm system according to some embodiments.

[0058] FIG. 20 illustrates steps of a method for operating a robot arm system according to some embodiments.

[0059] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The following description is intended only to illustrate the embodiments and is not intended to limit or restrict the scope of the rights of the present invention. Anything that can be readily inferred from the detailed description and examples of the invention by those skilled in the art should be construed as falling within the scope of the present invention. Details widely known to those skilled in the art will be omitted for brevity.

[0060] The terms used in this invention are described as general terms widely used in the technical fields related to the invention. However, the meaning of the terms used in this invention may vary depending on the intentions of engineers working in the relevant field, the emergence of new technologies, examination standards, or precedents. Some terms may be arbitrarily selected by the applicant, in which case the meanings of such arbitrarily selected terms will be explained in detail. The terms used in this invention should be interpreted not simply based on their dictionary meanings, but rather based on their meanings reflecting the overall context of the specification.

[0061] FIG. 1 illustrates how a robot arm movement track optimization device according to some embodiments operates.

[0062] Referring to FIG. 1, a system (100) may be illustrated to illustrate how a robot arm movement track optimization device operates.

[0063] In the system (100), when a demonstrator of painting or decontamination work performs a work demonstration (110), a work image can be created by taking a picture of the work, and 6-DOF pose values ​​of the sprayer or work tool can be extracted from the work image. Based on this, a robot arm movement track can be created to automate the demonstrator's manual work, and an optimized pose trajectory (130) can be created by editing the robot arm movement track.

[0064] The robot arm movement track optimization device (120) can operate based on the generation of 6D path primitives using statistical techniques. The robot arm movement track optimization device (120) can preprocess time-series image data to extract very simple geometric elements (primitives), such as straight lines or arcs, as a result, and optimize the robot arm movement track based on these.

[0065] The robot arm movement track optimization device (120) may include a photographing means for photographing a demonstrator's work demonstration (110) and a processing means for processing the work image and generating an optimized pose trajectory (130). In addition to painting or decontamination work, other types of work that can be generated as a movement track of the robot arm by tracking the 3D position and 3D direction of a sprayer or work tool may also be applied to the system (100).

[0066] FIG. 2 illustrates elements constituting a robot arm movement track optimization device according to some embodiments.

[0067] Referring to FIG. 2, the robot arm movement track optimization device (120) may include a photographing unit (121) and a first processing unit (122). However, the present invention is not limited thereto, and some components may be omitted from the robot arm movement track optimization device (120), or other general-purpose components may be further included in the robot arm movement track optimization device (120).

[0068] The photographing unit (121) may include a camera or the like for photographing a demonstrator performing a work demonstration (110). For example, the photographing unit (121) may be a 3D camera capable of extracting the 3D position and 3D direction of a sprayer or other work tool during the work demonstration (110).

[0069] The first processing unit (122) may include a memory and a processor. The memory may be configured to store various commands, computer programs, software, mobile applications, or data processed in the robot arm movement track optimization device (120). For example, the memory may be implemented as a non-volatile memory such as ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, etc., or a volatile memory such as DRAM, SRAM, SDRAM, PRAM, RRAM, FeRAM, etc., and may be implemented in the form of an HDD, SSD, SD, Micro-SD, etc., or a combination thereof.

[0070] The processor may be configured to perform processing steps required for the operation of the robot arm movement track optimization device (120). The processor may be implemented as an array of multiple logic gates for processing various operations or as a general-purpose microprocessor, and may be comprised of a single processor or multiple processors. For example, the processor may be implemented in the form of at least one of a microprocessor, a CPU, a GPU, and an AP.

[0071] The photographing unit (121) may be configured to capture the state of the work demonstration (110) at regular intervals during the demonstrator's work demonstration (110) to generate time-series image data. For example, the state of the work demonstration (110), the position and direction of the sprayer or other work tool, etc. may be captured at intervals of 1 second, 0.5 seconds, 2 seconds, 3 seconds, 5 seconds, or any other numerical value. In some embodiments, the time interval for capturing the state of the work demonstration (110) may not be regular.

[0072] The first processing unit (122) may be configured to extract 6D pose values ​​of the work demonstration (110) from the time series image data to generate time series pose data. In some cases, AI techniques may be utilized to extract the 6D pose values. The position and orientation of the sprayer or other work tool may be extracted from each image of the time series image data, and various image analysis tools may be utilized for this purpose. The 3D position and 3D orientation extracted from each image may constitute a 6D pose value. The time series pose data may include 6D pose values. By connecting the 6D pose values, a trajectory in 3D space may be generated.

[0073] The first processing unit (122) can be trained to derive an error range of the time series pose data based on a statistical technique, and perform path smoothing on the time series pose data based on the error range to generate the robot arm pose trajectory.

[0074] The first processing unit (122) may be configured to generate a robot arm pose trajectory for automating a task demonstration (110) by performing trajectory simplification on time-series pose data. Since time-series pose data includes human errors of the demonstrator and unavoidable work inefficiencies due to the work environment, simplification of the trajectory of the six-dimensional pose values ​​may be performed to optimize the painting / decontamination work. For example, trajectory simplification may include trajectory smoothing, etc.

[0075] The first processing unit (122) may be configured to edit duplicate data in the robot arm pose trajectory to generate an optimized pose trajectory (130). Since the robot arm pose trajectory may include the time the demonstrator stays in a specific area or the time the demonstrator pauses and waits, this may result in inefficiencies. To address this, editing of duplicate data may be performed.

[0076] According to an embodiment, the first processing unit (122) may be configured to derive an error range of time-series pose data and perform path smoothing on the time-series pose data based on the error range to generate a robot arm pose trajectory. The error range and path smoothing may be referred to FIG. 5, which will be described later. An error trajectory fitting a demonstrator's movement path may be derived, and a straight path and / or a curved path of the robot arm pose trajectory may be derived within the error trajectory. For example, the data may be converted into coordinate systems such as an orthogonal coordinate system, a cylindrical coordinate system, a spherical coordinate system, etc., and the data distribution on each coordinate system may be statistically analyzed, and a trajectory having the highest statistical correlation among primitive trajectories such as a straight trajectory, a circular / elliptical trajectory, and a spring trajectory may be utilized. The time-series pose data may be converted into the form of primitive data together with major parameters (start point, end point, curvature, cylindrical pitch, rate of change of direction vector, etc.).

[0077] In an embodiment, the work demonstration (110) may include a surface painting operation on a 3D object, and the 6D pose values ​​may include 3D position values ​​and 3D direction values ​​of a spray painting tool. In order for the robot arm to replace a worker for painting / decontamination, etc., the position and direction in which the worker sprays paint or decontamination agent on the 3D object may be extracted as 6D pose values. For example, the surface painting operation may include surface painting of a large ship, decontamination of nuclear power plant dismantling parts, etc.

[0078] According to an embodiment, the first processing unit (122) may be configured to perform path smoothing based on the 3D movement of the sprayed object sprayed by the spray painting mechanism, the spray amount of the sprayed object, the work completion rate status of the work demonstration, the work time limit of the work demonstration, the shape of the 3D object, and weather factors. For example, if there is a distance condition that must be kept away from the 3D object according to the specifications of the paint / decontaminant sprayer, path smoothing may be performed by reflecting this.

[0079] According to an embodiment, the first processing unit (122) may be configured to detect a stop section of the demonstrator based on time series pose data, and generate a robot arm pose trajectory by dividing the time series pose data into different paths based on the stop section. For the path division by the stop section, reference may be made to FIGS. 5, 8, and 9, which will be described later. If there is a section where the demonstrator stops for a certain period of time during the work demonstration (110), the robot arm may also perform additional work after stopping at the section. To this end, the movement paths of the robot arm pose trajectory may be divided based on the stop section.

[0080] According to an embodiment, the first processing unit (122) may be configured to generate a robot arm pose trajectory by combining a stationary section, a straight path section, and a curved path section. Although the demonstrator's work demonstration (110) is difficult to express as a straight line or a curved path, straight path sections and curved path sections can be derived through data fitting after path smoothing. Through such simplification, the learning process or execution process of the robot arm control model can be optimized. If it is determined that it is necessary to stay and work on a specific part of the 3D object for a long time, a stationary section of the robot arm pose trajectory can be set for the part.

[0081] According to an embodiment, the first processing unit (122) may be configured to generate a straight path segment by combining a first type for non-rotational straight movement, a second type for rotational straight movement, and a third type for non-rotational pure rotation, and to generate a curved path segment by combining a fourth type for non-rotational curved movement, a fifth type for rotational curved movement, and a sixth type for rotational spiral movement. For the types for generating the straight path segment, reference may be made to FIG. 6, which will be described later, and for the types for generating the curved path segment, reference may be made to FIG. 7, which will be described later. A combination of the first to third types for combinations of straight movement and rotation may be utilized to simplify generation of the straight path segment. Similarly, a combination of the fourth to sixth types may be utilized to simplify generation of the curved path segment.

[0082] According to an embodiment, the first processing unit (122) may be configured to generate segment parameters of a stationary segment, a straight path segment, and a curved path segment, wherein the first segment parameter of the stationary segment may include a stationary time, the second segment parameter of the straight path segment may include a start point, an end point, and a progress speed, and the third segment parameter of the curved path segment may include a start point, an end point, a progress speed, and a curvature radius. With respect to the segment parameters, reference may be made to FIG. 9, which will be described later. Adjustment of the segment parameters may be performed when tuning of the robot arm pose trajectory is required.

[0083] In an embodiment, the first processing unit (122) may be configured to generate an optimized pose trajectory by deleting at least some of the stop sections connecting paths of the robot arm pose trajectory. The straight / curved path sections of the robot arm pose trajectory may be connected by stop sections, and each stop section may be classified as either an intentionally generated stop section or an unnecessary stop section.

[0084] According to an embodiment, the first processing unit (122) may be configured to classify the stop sections into first stop sections where the demonstrator intentionally stops and unnecessary second stop sections, and delete the second stop sections to generate an optimized pose trajectory. The necessity of each stop section may be classified, and unnecessary stop sections may be deleted. For example, whether each stop section is necessary may be determined based on the surface shape of the 3D object. A stop section formed on a surface shape with a simple structure may be classified as unnecessary.

[0085] According to an embodiment, the first processing unit (122) may be configured to calculate a first learning required time of a robot arm control model for time series pose data before movement track optimization is performed and a second learning required time of a robot arm control model for an optimized pose trajectory (130) after movement track optimization is performed, and provide a comparison result of the first learning required time and the second learning required time. After the optimized pose trajectory (130) is generated, the robot arm control model may be trained based on the optimized pose trajectory (130), and the operation of the robot arm may be controlled through the trained robot arm control model. In this case, the first learning required time and the second learning required time may be compared to evaluate the performance of the optimized pose trajectory (130). Additionally, the time required for a task on a 3D object may be compared before and after generation of the optimized pose trajectory (130) to evaluate the performance of the optimized pose trajectory (130).

[0086] FIG. 3 illustrates a process of generating a movement track of a robot arm by capturing a demonstrator's demonstration motion for a 3D object according to some embodiments.

[0087] Referring to FIG. 3, a demonstrator (310) can demonstrate a painting / decontamination operation on a 3D object (320), and an optimized pose trajectory (340) can be generated based on this.

[0088] The robot arm (330) can perform painting / decontamination work on a 3D object (320) on behalf of the demonstrator (310) based on the optimized pose trajectory (340). For example, the 3D object (320) may include a large ship, nuclear power plant waste parts generated during nuclear power plant dismantling, etc. Since the 3D object (320) may have a complex, irregular structure, the optimized pose trajectory (340) can be generated based on the work demonstration (110) of the demonstrator (310).

[0089] FIG. 4 illustrates a three-dimensional position and three-dimensional orientation of a robot arm according to some embodiments.

[0090] Referring to FIG. 4, the three-dimensional position (420) and three-dimensional direction (430) of the robot arm (410) can be illustrated.

[0091] The 3D position (420) and 3D direction (430) can constitute 6D pose values ​​of the robot arm (410). The 6D pose values ​​can be extracted by the robot arm movement track optimization device (120), and an optimized pose trajectory (130) can be generated through trajectory simplification and duplicate editing.

[0092] FIG. 5 illustrates a process for generating a robot arm pose trajectory according to some embodiments.

[0093] Referring to FIG. 5, an image (500) illustrating a process of generating a robot arm pose trajectory can be illustrated.

[0094] Referring to image (500), the 6D pose of the demonstrator can be periodically calculated and plotted in 3D space. The 6D pose of the demonstrator without trajectory simplification / smoothing may have various deviations, such as the demonstrator's erroneous movements, hand tremors, or errors due to the work environment. These deviations can be calculated in the form of an error / error range, and a smoothed curved path can be derived within the error / error range.

[0095] In image (500), two curved path trajectories can be connected by a single stationary section. A region where a large number of values ​​of the demonstrator's 6D pose are concentrated may indicate that the demonstrator has remained for a long period of time, and this can be treated as a stationary section. Based on the surface structure complexity of the 3D object in the corresponding section, the stationary section can be classified as unnecessary or unavoidably necessary.

[0096] FIG. 6 illustrates a process for generating a straight path section of a robot arm pose trajectory according to some embodiments.

[0097] Referring to FIG. 6, an image (600) illustrating a process of generating a straight path section of a robot arm pose trajectory can be illustrated.

[0098] A straight path section can be generated by combining a first type (type 1-1) for non-rotational straight movement, a second type (type 1-2) for rotational straight movement, and a third type (type 1-3) for non-rotational pure rotation. In the image (600), the first type (type 1-1) can be represented as section 1, and the second type (type 1-2) can be represented as section 2.

[0099] For the first type (type 1-1), the 3D position p(t) can be changed by the velocity v(t), but the 3D direction q(t) can be kept constant. For the second type (type 1-2), the 3D position p(t) can be changed by the velocity v(t), and the 3D direction q(t) can be changed by a constant rotation R(ωt). For the third type (type 1-3), the 3D position p(t) can be kept constant, and the 3D direction q(t) can be changed by a constant rotation R(ωt).

[0100] FIG. 7 illustrates a process of generating a curved path section of a robot arm pose trajectory according to some embodiments.

[0101] Referring to FIG. 6, an image (700) illustrating a process of generating a curved path section of a robot arm pose trajectory can be illustrated.

[0102] A curved path section can be created by combining the fourth type (type 2-1) for non-rotating curved movement, the fifth type (type 2-2) for rotating curved movement, and the sixth type (type 2-3) for rotating spiral movement.

[0103] For the fourth type (type 2-1), the 3D position p(t) can be changed by a constant rotation R(ωt), but the 3D direction q(t) can be kept constant. For the fifth type (type 2-2), the 3D position p(t) can be changed by a constant rotation R(ωt), and the 3D direction q(t) can also be changed by the constant rotation R(ωt). For the sixth type (type 2-3), the 3D position p(t) can be changed spirally by a constant rotation R(ωt) and velocity v(t), and the 3D direction q(t) can be changed by the constant rotation R(ωt).

[0104] Figure 8 illustrates a process for performing trajectory simplification on time series pose data according to some embodiments.

[0105] Referring to FIG. 8, time series pose data (810) may be source data for a 6D path, and robot arm pose trajectory (820) may be segmentation into raw instances. As in table (830), the raw instances may be classified into straight lines, curves (arcs), and waits.

[0106] The time series pose data (810) may refer to 6D pose values ​​extracted from the demonstrator's work demonstration (110), which may have various deviations such as errors due to the demonstrator's erroneous movements, hand tremors, or work environment. Thereafter, the time series pose data (810) may be simplified / smoothed into a robot arm pose trajectory (820).

[0107] Table (830) can represent straight path sections, curved path sections, and stationary sections that constitute the robot arm pose trajectory (820). Sections ①, ③, ④, ⑤, and ⑥ can be straight path sections. Section ② can be a curved path section. The section before section ①, between sections ② and ③, and between sections ⑤ and ⑥ can be stationary sections. As in table (830), section ① can be composed of the 7th data (p7) to the 15th data (p15) among the time series pose data (810).

[0108] FIG. 9 illustrates a process of creating an optimized pose trajectory by editing duplicate data in a robot arm pose trajectory according to some embodiments.

[0109] Referring to FIG. 9, the robot arm pose trajectory (910) may be identical to the robot arm pose trajectory (820), and a stationary section (920) may be identified among the robot arm pose trajectories (820).

[0110] The table (900) may represent three stop sections (seq 1, seq 5, seq 8) among the robot arm pose trajectory (820). The three stop sections (seq 1, seq 5, seq 8) may be classified as essential or not, and unnecessary stop sections may be deleted. For example, the stop section of seq 1 may be classified as corresponding to a waiting state before the demonstrator begins the task demonstration (110), and may be deleted from the robot arm pose trajectory (120) during the process of generating the optimized pose trajectory (130).

[0111] Considering the nature of the pause interval, the pause interval of seq 5 may not be deleted, while the pause interval of seq 8 may be deleted. For example, seq 5 may be determined to be an intentional pause by the demonstrator to continue the injection operation, but seq 8 may be determined to be an unnecessary pause interval.

[0112] FIG. 10 illustrates steps of a method for optimizing a robot arm movement track according to some embodiments.

[0113] Referring to FIG. 10, a method (1000) for optimizing a robot arm movement track based on generating 6D path primitives using statistical techniques may include steps (1010) to (1040). However, the method is not limited thereto, and some steps may be omitted or other general steps may be added, and the steps of the method (1000) for optimizing a robot arm movement track may be executed in a different order than the illustrated order.

[0114] The robot arm movement track optimization method (1000) may be composed of steps that are processed in a time-series manner in the robot arm movement track optimization device (120). Therefore, even if the content is omitted below, the content described above for the robot arm movement track optimization device (120) may be equally applied to the robot arm movement track optimization method (1000).

[0115] Steps (1010) to (1040) of the robot arm movement track optimization method (1000) can be performed by the photographing unit (121) and the first processing unit (122) of the robot arm movement track optimization device (120).

[0116] In step (1010), the robot arm movement track optimization device (120) can perform a step of generating time series image data by photographing the state of the work demonstration at regular intervals during the work demonstration of the demonstrator through a photographing unit.

[0117] In step (1020), the robot arm movement track optimization device (120) can perform a step of generating time series pose data by extracting 6D pose values ​​of a task demonstration from time series image data through a first processing unit.

[0118] In step (1030), the robot arm movement track optimization device (120) may perform a step of generating a robot arm pose trajectory for automating a task demonstration by performing trajectory simplification on time series pose data through a first processing unit.

[0119] In step (1040), the robot arm movement track optimization device (120) can perform a step of editing duplicate data in the robot arm pose trajectory through the first processing unit to generate an optimized pose trajectory.

[0120] According to an embodiment, the method for optimizing a robot arm movement track (1000) may be implemented in the form of a computer program stored on a computer-readable storage medium. That is, the computer program may include instructions for implementing the method for optimizing a robot arm movement track (1000), and the instructions of the program may be stored on a computer-readable storage medium. The computer program may include a mobile application.

[0121] According to an embodiment, the computer-readable storage medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute computer program instructions such as ROMs, RAMs, flash memories, and the like. The computer program instructions may include machine language codes generated by a compiler and high-level language codes that can be executed by a computer using an interpreter, etc.

[0122] FIG. 11 illustrates how a pose difference recognition-based painting device operates according to some embodiments.

[0123] Referring to FIG. 11, a system (1100) that automates a painting operation on a 3D object (1110) using a pose difference recognition-based painting device (1120) to generate a painting result (1130) can be illustrated.

[0124] The painting operation of the painting device (1120) based on pose difference recognition can be performed using a robot arm movement track. The painting operation can be performed by spraying paint onto an object. The spraying of the paint can be performed by a robot arm. The robot arm can change the spraying direction of the spray gun toward a predetermined direction at each point of the track while moving along the robot arm movement track. Each point of the track can have a 3D position and a 3D direction, and the 3D position and the 3D direction can constitute a 6D pose.

[0125] The 3D object (1110) may be one of multiple objects having the same shape. A representative object may be selected as a sample to automate the painting process for multiple objects. An automated painting process for the sample object may be established, which may then be applied to other objects after correction. Therefore, even without generating robot arm movement tracks for all of the objects, it may be possible to apply the process to multiple objects through track correction.

[0126] FIG. 12 illustrates elements constituting a painting device based on pose difference recognition according to some embodiments.

[0127] Referring to FIG. 12, a pose difference recognition-based painting device (1120) may include an AI pose extraction unit (1121), a second processing unit (1122), and a work unit (1123). However, the present invention is not limited thereto, and some components may be omitted from the pose difference recognition-based painting device (1120), or other general-purpose components may be further included in the pose difference recognition-based painting device (1120).

[0128] The AI ​​pose extraction unit (1121) may include a camera or the like for capturing a 3D object (1110). For example, the AI ​​pose extraction unit (1121) may capture a 3D object (1110) when the 3D object (1110) activates a capture trigger in a capture area. The AI ​​pose extraction unit (1121) may be a 3D camera capable of extracting a 3D position and 3D direction of the 3D object (1110).

[0129] The second processing unit (1122) may include a memory and a processor. The memory may be configured to store various commands, computer programs, software, mobile applications, or data processed in the pose difference recognition-based painting device (1120). For example, the memory may be implemented as a non-volatile memory such as ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, etc., or a volatile memory such as DRAM, SRAM, SDRAM, PRAM, RRAM, FeRAM, etc., and may be implemented in the form of an HDD, SSD, SD, Micro-SD, etc., or a combination thereof.

[0130] The processor may be configured to perform the processing steps required for the operation of the pose difference recognition-based painting device (1120). The processor may be implemented as an array of multiple logic gates for processing various operations or as a general-purpose microprocessor, and may be comprised of a single processor or multiple processors. For example, the processor may be implemented in the form of at least one of a microprocessor, a CPU, a GPU, and an AP.

[0131] The AI ​​pose extraction unit (1121) may be configured to capture a 3D object (1110) to generate a target image. The AI ​​pose extraction unit (1121) may capture the 3D object (1110) when it is moved to a capture area. The AI ​​pose extraction unit (1121) may capture the 3D object (1110) when the 3D object (1110) activates a capture trigger in the capture area. For example, the capture trigger may include the 3D object (1110) passing through a detection sensor.

[0132] The AI ​​pose extraction unit (1121) may be configured to extract 6D pose values ​​of a 3D object (1110) from a target image. Extraction of the 6D pose values ​​may be performed based on an AI model. For example, the AI ​​model may include a neural network model trained through various machine learning techniques. The 6D pose values ​​may include 3D position values ​​and 3D direction values. Extraction of the 6D pose values ​​may be performed through various image processing tools. For example, the 6D pose values ​​may be extracted by calculating the difference from a reference position and a reference direction. Specifically, the AI ​​model for extracting the 6D pose values ​​may be trained using existing 3D CAD information, or may be trained by taking one or more 2D images and then extracting pose information of an object taken on site. Meanwhile, the AI ​​model may be generated based on existing computer vision techniques, such as surface matching, rather than deep learning.

[0133] The second processing unit (1122) may be configured to calculate a pose difference value between a sample pose value prepared in advance and a 6D pose value based on a sample object. The sample object may be an object having the same shape as the 3D object (1110). Since the 3D position and 3D direction of the sample object in the process of generating the sample pose value may be different from the 3D position and 3D direction captured for the 3D object (1110), a pose difference value corresponding to the difference may be calculated. The pose difference value may include a position difference value and a direction difference value.

[0134] The second processing unit (1122) may be configured to correct a sample robot arm movement track prepared in advance for a sample pose value based on a pose difference value into a corrected robot arm movement track for a 6D pose value of the 3D object (1110). Since the sample robot arm movement track is generated based on the sample pose value, if the 6D pose value of the 3D object (1110) is different from the sample pose value, it may be difficult to directly apply the sample robot arm movement track, and correction may be performed to resolve this difference. According to an embodiment, an AI correction model may be utilized for the correction. For example, the AI ​​correction model may include a neural network model learned through various machine learning techniques.

[0135] The work unit (1123) may be configured to perform a painting operation on a 3D object (1110) along a calibration robot arm movement track. The work unit (1123) may include a robot arm. The robot arm of the work unit (1123) may move along 3D position values ​​of the calibration robot arm movement track. The robot arm may spray a painting material toward a 3D direction value at each 3D position value. The work unit (1123) may perform a painting operation when the 3D object (1110) activates a work trigger in the work area. For example, the work trigger may include the 3D object (1110) being detected by a detection sensor.

[0136] According to an embodiment, the pose difference recognition-based painting device (1120) may further include a transport unit configured to transport the 3D object (1110) so as to sequentially pass through a photographing area, a buffering area, and a painting area. The transport unit may include a conveyor belt or other transport means. Reference may be made to FIG. 3, which will be described later, regarding the photographing area, the buffering area, and the painting area.

[0137] According to an embodiment, the AI ​​pose extraction unit (1121) may be configured to generate a target image when the 3D object (1110) activates a shooting trigger in a shooting area, and the work unit (1123) may be configured to perform a painting operation when the 3D object (1110) activates a work trigger in a work area. For example, the shooting trigger and / or the work trigger may include the 3D object (1110) entering a specific position by a transport unit.

[0138] In an embodiment, the buffer zone may be configured to reduce dust generated by the painting work in the painting area, and the second processing unit (1122) may be configured to generate a correction robot arm movement track while the 3D object (1110) passes through the buffer zone before reaching the painting area. Since dust may be generated due to the painting paint, etc. when the painting work is performed in the painting area, the buffer zone may be formed to protect the photographing area. The size of the buffer zone may be changed depending on the dust prevention performance. Preparation for the painting work on the 3D object (1110) may be completed while passing through the buffer zone. Preparation for the painting work may include generating a correction robot arm movement track. In an embodiment, the second processing unit (1122) may be configured to generate the correction robot arm movement track in real time immediately after the pose difference value is calculated by the AI ​​pose extraction unit (1121).

[0139] According to an embodiment, the sample object may be configured as a digital twin having the same shape as the 3D object (1110), and the second processing unit (1122) may be configured to compare a sample 6D pose value of the digital twin with a 6D pose value of the 3D object to calculate a pose difference value. The digital twin may be digital data reflecting the same size, shape, material properties, etc. as an actual object. The digital twin of the sample object may have a fixed sample 6D pose value, which may be compared with a 6D pose value that is newly measured each time for a different 3D object (1110).

[0140] According to an embodiment, the second processing unit (1122) may be configured to compare the 3D position value of the digital twin with the 3D position value of the 3D object (1110) to generate a position difference value, compare the 3D direction value of the digital twin with the 3D direction value of the 3D object (1110) to generate a direction difference value, and calculate a pose difference value based on the position difference value and the direction difference value. For example, each 3D position and position difference value may be expressed as a 3D vector. Additionally, each 3D direction and direction difference value may be expressed as a 3D vector. For the purpose of computational efficiency, etc., the position difference value and the direction difference value may be handled separately.

[0141] According to an embodiment, the 3D object (1110) may include a 2D object hanging on a wall, and the position difference value may include a height difference value between the height of the 2D object and the height of the digital twin, and the direction difference value may include an angular difference value between the angle of the 2D object and the angle of the digital twin. For example, the 2D object may include a picture frame. Since the 6D pose value of a 2D object such as a picture frame can be expressed as a height value hanging on a wall and an angle value hanging on a wall, it can have high computational efficiency compared to other space-based objects.

[0142] According to an embodiment, the second processing unit may be configured to generate a corrected robot arm movement track by applying a 3D positional translation and a 3D orientation rotation corresponding to a pose difference value to the sample robot arm movement track. The positional difference value may be reflected by the 3D positional translation, and the orientation difference value may be reflected by the 3D orientation rotation. Since the positional difference value and the orientation difference value may be reflected separately, computational efficiency may be improved compared to composite computation.

[0143] According to an embodiment, the AI ​​correction model may be trained to select a first sample 6D pose value that is closest to the 6D pose value of a 3D object (1110) among a plurality of sample 6D pose values ​​prepared in advance at intervals of a certain distance and a certain angle, and to set a robot arm movement track corresponding to the first sample 6D pose value as a correction robot arm movement track. For example, in the case of a 2D object such as a picture frame hanging on a wall, multiple (11*11 = 121) sample 6D pose values ​​can be prepared in advance at 1 cm intervals (e.g., -5, -4, -3, ..., +3, +4, +5) and 5 degree intervals (e.g., -25 degrees, -20 degrees, -15 degrees, ..., +15 degrees, +20 degrees, +25 degrees), and among the 121 sample 6D pose values, the sample 6D pose value closest to the 6D pose value of the 3D object (1110) can be selected and utilized.

[0144] With respect to the sample 6D pose value, reference path data (Fig. 15) can be prepared for an initial single reference sample based on a method such as path teaching based on human motion data, path teaching by manual robot operation, path generation based on 3D CAD data, path generation based on 2D image (Fig. 14), etc. The second processing unit can capture 2D or 3D images for other samples of the same shape, derive the 6D pose value of the current sample based on the 2D / 3D image information (Fig. 16), derive a conversion formula for converting the 6D pose value of the reference sample to the 6D pose value of the current sample, and apply the conversion formula to the reference path data to derive the path data for the current sample (Fig. 17).

[0145] Since the robot arm of the work unit (1123) sprays the paint in a spray form, the painting process can be performed smoothly even if there are some discrepancies or tolerances in the 6D pose. To utilize this, multiple sample 6D pose values ​​prepared in advance can be utilized. The process of selecting the closest sample 6D pose value can be performed by an AI correction model.

[0146] In an embodiment, the AI ​​correction model may be configured to generate a sample robot arm movement track by projecting a 2D path onto an offset surface of a sample object. Referring to FIG. 14, which will be described later, for a method of generating the robot arm movement track. Since the sample object may have a complex 3D surface structure, an offset surface and projection may be utilized to smoothly reflect the 2D path onto the complex 3D surface structure.

[0147] FIG. 13 illustrates a process of performing a painting operation on a 3D object according to some embodiments.

[0148] Referring to FIG. 13, a flow (1120) illustrating a process of performing a painting operation on a 3D object (1110) may be illustrated. A transport device (1124) may transport the 3D object (1110).

[0149] A 3D object (1110) can sequentially pass through a shooting zone, a buffer zone, and a work zone. When a shooting trigger is activated in the shooting zone, an AI pose extraction unit (1121) can operate, and when a work trigger is activated in the work zone, a work unit (1123) can operate. In the buffer zone, operations of a second processing unit (1122) can be performed. When all zones are passed, a painting result (1130) corresponding to the 3D object (1110) can be generated without human intervention through robot automation.

[0150] Figure 14 illustrates a process for generating a robot arm movement track according to some embodiments.

[0151] Referring to FIG. 14, an image (1400) illustrating a process of creating a robot arm movement track may be illustrated.

[0152] A 2D path (1410) can be selected as the most suitable path for a 3D object (1110) or a sample object. An offset surface (1420) can be extracted for the 3D object (1110), and a projection (1430) can be performed on it. A robot arm movement track (1440) can be generated as a result of the projection (1430) on the offset surface (1420).

[0153] At this time, the position with the widest projection area of ​​the offset surface (1420) is set as the reference area, and the amount of sprayed paint is increased in inverse proportion to the amount that the projected offset surface (1420) area is reduced compared to the reference area, or conversely, the movement speed of the 3D object or spray gun is proportionally slowed down so that a flawless painting operation can be performed.

[0154] Fig. 15 illustrates a specific form of a robot arm movement track according to some embodiments.

[0155] Referring to FIG. 15, a specific form of a robot arm movement track (1500) may be illustrated. For example, the 3D object may be a picture frame hanging on a wall.

[0156] The robot arm movement track (1500) can move along the frame perimeter of the picture frame. While moving, the robot arm movement track (1500) can have a zigzag path. In this way, the efficiency of the work unit (1123) in painting the 3D object can be improved. Meanwhile, the width or front-to-back spacing of the zigzag path can be adjusted based on the characteristics of the 3D object and / or the characteristics of the spray material.

[0157] Figure 16 illustrates a process for calculating a pose difference value for a sample pose value according to some embodiments.

[0158] Referring to FIG. 16, the process of calculating a pose difference value for a sample pose value may include a process of calculating a position difference value and a process of calculating a direction difference value.

[0159] In the process of calculating the position difference value, the difference between the 3D position value of the sample object (1611) and the 3D position value of the 3D object (1612) to be painted can be calculated. Similarly, in the process of calculating the direction difference value, the difference between the 3D direction value of the sample object (1621) and the 3D direction value of the 3D object (1622) to be painted can be calculated. The AI ​​correction model can convert the sample robot arm movement track into a correction robot arm movement track by separately applying the position difference value and the direction difference value.

[0160] FIG. 17 illustrates a process of correcting a sample robot arm movement track to a corrected robot arm movement track according to some embodiments.

[0161] Referring to FIG. 17, a process of calibrating a sample robot arm movement track (1710) to a correction robot arm movement track (1720) may be illustrated. For example, the calibration process may be performed using an AI calibration model.

[0162] The sample robot arm movement track (1710) can be pre-generated and prepared for the sample object, and the sample robot arm movement track (1710) can be calibrated and utilized for a 3D object (1110) having the same structure / shape as the sample object. Calibration with the calibration robot arm movement track (1720) can include positional correction and orientation correction.

[0163] By position correction, the 3D position of the sample robot arm movement track (1710) can be changed by an amount corresponding to the position difference between the sample object and the 3D object (1110) (see 1611, 1612 of FIG. 16). In addition, by direction correction, the 3D direction of the sample robot arm movement track (1710) can be changed by an amount corresponding to the direction difference between the sample object and the 3D object (1110) (see 1621, 1622 of FIG. 16). As a result, a corrected robot arm movement track (1720) can be generated. As another example, among a plurality of sample 6D pose values ​​prepared in advance, a sample 6D pose value closest to a 6D pose value measured for the 3D object (1110) can be selected.

[0164] Figure 18 illustrates steps of a painting method based on pose difference recognition according to some embodiments.

[0165] Referring to FIG. 18, the painting method (1800) based on pose difference recognition may include steps (1810) to (1850). However, the present invention is not limited thereto, and some steps may be omitted or other general steps may be added, and the steps of the painting method (1800) based on pose difference recognition may be executed in a different order than the illustrated order.

[0166] The painting method (1800) based on pose difference recognition may be composed of steps that are processed in a time-series manner in the painting device (1120) based on pose difference recognition. Therefore, even if the content is omitted below, the content described above for the painting device (1120) based on pose difference recognition may be equally applied to the painting method (1800) based on pose difference recognition.

[0167] Steps (1810) to (1850) of the painting method (1800) based on pose difference recognition can be performed by the AI ​​pose extraction unit (1121), the second processing unit (1122), and the work unit (1123) of the painting device (1120) based on pose difference recognition.

[0168] In step (1810), a pose difference recognition-based painting device (1120) can perform a step of generating a target image by photographing a 3D target through an AI pose extraction unit.

[0169] In step (1820), the pose difference recognition-based painting device (1120) can perform a step of extracting a 6D pose value of a 3D object from a target image through an AI pose extraction unit.

[0170] In step (1830), the painting device (1120) based on pose difference recognition can perform a step of calculating a pose difference value between a sample pose value and a 6D pose value prepared in advance based on a sample object through a second processing unit.

[0171] In step (1840), the painting device (1120) based on pose difference recognition can perform a step of correcting a sample robot arm movement track prepared in advance for a sample pose value based on a pose difference value through a second processing unit into a corrected robot arm movement track for a 6D pose value of a 3D object.

[0172] At step (1850), the painting device (1120) based on pose difference recognition can perform a step of performing a painting task on a 3D object along a correction robot arm movement track through a work unit.

[0173] According to an embodiment, the painting method (1800) based on pose difference recognition may be implemented in the form of a computer program stored on a computer-readable storage medium. That is, the computer program may include commands for implementing the painting method (1800) based on pose difference recognition, and the program commands may be stored on a computer-readable storage medium. The computer program may include a mobile application.

[0174] According to an embodiment, the computer-readable storage medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute computer program instructions such as ROMs, RAMs, flash memories, and the like. The computer program instructions may include machine language codes generated by a compiler and high-level language codes that can be executed by a computer using an interpreter, etc.

[0175] FIG. 19 illustrates elements constituting a robot arm system according to some embodiments.

[0176] Referring to FIG. 19, the robot arm system (1900) may include a robot arm movement track optimization device (120) and a painting device (1120). However, the present invention is not limited thereto, and some components may be omitted from the robot arm system (1900), or other components may be further included in the robot arm system (1900).

[0177] A robot arm movement track optimization device (120) based on 6D path primitive generation using statistical techniques can be configured to generate an optimized pose trajectory. A painting device (1120) based on pose difference recognition using artificial intelligence (AI) can be configured to perform a painting task by treating the optimized pose trajectory as a sample robot arm movement track.

[0178] FIG. 20 illustrates steps of a method for operating a robot arm system according to some embodiments.

[0179] Referring to FIG. 20, the method (2000) for operating a robot arm system may include steps (2010) and (2020). However, the present invention is not limited thereto, and some steps may be omitted or other steps may be added, and steps (2010) and (2020) may be performed in a different order than the illustrated order. The method (2000) for operating a robot arm system may be performed by the robot arm system (1900).

[0180] In step (2010), the robot arm system (1900) may perform a step of generating an optimized pose trajectory using a robot arm movement track optimization device based on 6D path primitive generation using a statistical technique. In step (2020), the robot arm system (1900) may perform a step of performing a painting task by treating the optimized pose trajectory as a sample robot arm movement track using a painting device based on pose difference recognition using artificial intelligence (AI).

[0181] Although the embodiments of the present invention have been described in detail above, the scope of rights according to the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention described in the following claims should also be interpreted as being included in the scope of rights according to the present invention.

Claims

1. In the robot arm system, A robot arm movement track optimization device based on 6D path primitive generation utilizing statistical techniques, configured to generate an optimized pose trajectory; A robotic arm system comprising:

2. In paragraph 1, The above robot arm movement track optimization device is, A photographing unit configured to photograph the state of the work demonstration during the demonstrator's work demonstration to generate time series image data; and A robot arm system comprising a first processing unit that extracts 6D pose values ​​of the task demonstration from the time series image data to generate time series pose data, simplifies a trajectory for the time series pose data to generate a robot arm trajectory, and edits duplicate data to generate the optimized pose trajectory.

3. In paragraph 2, A robot arm system, wherein the first processing unit is configured to derive an error range of the time series pose data and perform path smoothing on the time series pose data based on the error range to generate the robot arm pose trajectory.

4. In paragraph 3, The above work demonstration includes a surface painting operation for a 3D object, and the 6D pose values ​​include 3D position values ​​and 3D direction values ​​of the spray painting tool, a robotic arm system.

5. In paragraph 4, A robot arm system, wherein the first processing unit is configured to perform the path smoothing based on the 3D movement of the spray body sprayed by the spray painting mechanism, the spray amount of the spray body, the work completion rate status of the work demonstration, and the work limit time of the work demonstration.

6. In paragraph 2, A robot arm system, wherein the first processing unit is configured to detect a stationary section of the demonstrator based on the time series pose data, and generate the robot arm pose trajectory by dividing the time series pose data into different paths based on the stationary section.

7. In paragraph 6, A robot arm system, wherein the first processing unit is configured to generate the robot arm pose trajectory as a combination of the stationary section, the straight path section, and the curved path section.

8. In paragraph 7, A robot arm system, wherein the first processing unit is configured to generate the straight path section by combining a first type of non-rotational straight movement, a second type of rotational straight movement, and a third type of non-rotational pure rotation, and to generate the curved path section by combining a fourth type of non-rotational curved movement, a fifth type of rotational curved movement, and a sixth type of rotational spiral movement.

9. In paragraph 7, The first processing unit is configured to generate section parameters of the stationary section, the straight path section, and the curved path section, A robot arm system, wherein the first section parameter of the above-mentioned stop section includes a stop time, the second section parameter of the above-mentioned straight path section includes a start point, an end point, and a progress speed, and the third section parameter of the above-mentioned curved path section includes a start point, an end point, a progress speed, and a curvature radius. 1st processing unit 10. In paragraph 8, A robot arm system, wherein the first processing unit is configured to classify the stop sections into first stop sections where the demonstrator intentionally stopped and unnecessary second stop sections, and delete the second stop sections to generate the optimized pose trajectory.

11. In paragraph 1, A painting device based on pose difference recognition utilizing artificial intelligence (AI), configured to perform painting work by treating the above-mentioned optimized pose trajectory as a sample robot arm movement track; The above painting device, An AI pose extraction unit configured to capture a 3D object to generate a target image and extract a 6D pose value of the 3D object from the target image; A second processing unit configured to calculate a pose difference value between a sample pose value prepared in advance based on a sample object and the 6D pose value, and to correct the sample robot arm movement track prepared in advance for the sample pose value based on the pose difference value to a correction robot arm movement track for the 6D pose value of the 3D object; and A robot arm system comprising a work unit configured to perform a painting operation on the 3D object along the above-described correction robot arm movement track.

12. In paragraph 11, The above painting device, A robot arm system further comprising a transport unit configured to transport the 3D object so as to sequentially pass through a shooting area, a buffer area, and a painting area.

13. In paragraph 12, The above AI pose extraction unit is configured to generate the target image when the 3D target activates a shooting trigger in the shooting area, A robotic arm system, wherein the above work unit is configured to perform the painting work when the 3D object activates a work trigger in the work area.

14. In paragraph 12, The above buffer zone is configured to reduce dust generated by the painting work in the above painting zone, A robot arm system, wherein the second processing unit is configured to generate the correction robot arm movement track while the 3D object passes through the buffer zone before reaching the painting zone.

15. In paragraph 12, A robot arm system, wherein the second processing unit is configured to generate the corrected robot arm movement track in real time immediately after the pose difference value is calculated by the AI ​​pose extraction unit.

16. In paragraph 11, The above sample object is composed of a digital twin having the same shape as the 3D object, A robot arm system, wherein the second processing unit is configured to compare a sample 6D pose value of the digital twin with the 6D pose value of the 3D object to calculate the pose difference value.

17. In paragraph 16, The second processing unit generates a position difference value by comparing the 3D position value of the digital twin with the 3D position value of the 3D object, Comparing the 3D direction value of the above digital twin with the 3D direction value of the above 3D object to generate a direction difference value, A robot arm system configured to calculate the pose difference value based on the position difference value and the direction difference value.

18. In paragraph 17, The above 3D object includes a 2D object hanging on a wall, The above position difference value includes a height difference value between the height of the 2D object and the height of the digital twin, A robot arm system, wherein the above direction difference value includes an angle difference value between the angle of the 2D object and the angle of the digital twin.

19. In paragraph 11, The second processing unit is configured to generate the corrected robot arm movement track by applying a 3D positional movement and a 3D direction rotation corresponding to the pose difference value to the sample robot arm movement track, A robot arm system in which the AI ​​correction model is trained to select a first sample 6D pose value closest to the 6D pose value of the 3D object from among a plurality of sample 6D pose values ​​prepared in advance at intervals of a certain distance and a certain angle, and to set the robot arm movement track corresponding to the first sample 6D pose value as the correction robot arm movement track.

20. In a method for operating a robot arm system, A step of generating an optimized pose trajectory using a robot arm movement track optimization device based on 6D path primitive generation using statistical techniques; and A method for driving a robot arm system, comprising: performing a painting task by treating the optimized pose trajectory as a sample robot arm movement track using a painting device based on pose difference recognition using artificial intelligence (AI);

Images