System and method for improving robot execution speed using spatial cognition-based demonstration data downsampling for imitation learning

Abstract

A system and method for improving the execution speed of a robot are provided, which can improve the execution speed of a robot by accurately distinguishing between precision sections and acceleration-capable sections by considering the semantic and spatial context of demonstration data and efficiently accelerating the entire data. A system for improving the execution speed of a robot according to one aspect of the present invention comprises: a memory for storing demonstration data of a robot; and by executing instructions stored in said memory, acquiring two-dimensional positions of an object and a gripper from video frames included in the demonstration data; calculating three-dimensional coordinates corresponding to said two-dimensional positions using a monocular depth estimation model; generating relationship information expressing changes in the spatial relationship between the object and the gripper by calculating the temporal rate of change and distance between said calculated three-dimensional coordinates; performing semantic analysis by inputting scene information acquired on a frame-by-frame basis along with said relationship information; classifying each segment of the demonstration data into a precision motion segment or a non-precision motion segment based on the result of said semantic analysis; applying a time-axis alignment algorithm to one of the classified segments as a reference demonstration segment to propagate the segment label of the reference demonstration segment to other demonstration segments of the demonstration data set; and downsampling according to said propagated label by maintaining the original density for the precision motion segment and expanding the frame interval for the non-precision motion segment, wherein the amount of movement of the robot after downsampling is equal to the amount of movement of the robot in said demonstration data and It includes a processor configured to generate training demonstration data adjusted to be identical.

Description

Technology Field

[0001] The present invention relates to a system and method for improving the execution speed of a robot, which improves the task execution speed of a robot through spatial recognition-based segment analysis and selective downsampling of demonstration data for robot imitation learning. Background Technology

[0002] Imitation learning-based robot control technology has rapidly advanced in the field of training robots to perform various manipulation tasks using human demonstration data. Recently, methods that learn policies for performing complex operations by combining video-based observations with continuous motion data have become mainstream; in particular, vision-robot policies based on behavior cloning are demonstrating high success rates in tasks such as complex object manipulation, scene recognition, and fine motion control. Unlike reinforcement learning, this technology does not require explicit reward design or online search processes, offering the advantages of high data efficiency and stability, and thus possesses the potential for widespread use in industrial and experimental environments.

[0003] However, the demonstration data that robots refer to during the learning process is mostly structured to be collected by humans directly performing manipulation tasks, and this process naturally tends to proceed with cautious and slow movements to ensure task accuracy. Humans reduce speed when approaching objects and repeat fine-tuning movements when performing precise steps such as alignment and insertion; furthermore, there are many situations where a constant speed must be maintained due to camera shooting conditions or equipment constraints. Consequently, these factors cause demonstration data to be recorded at a much slower speed than the actual speed at which the robot can perform the task, and the imitation learning policy learns these slow temporal patterns as they are, resulting in operation at the same speed during the execution phase.

[0004] Since policies learned by robots from demonstration data generally operate by mimicking the data's distribution, a problem arises where actual work efficiency decreases if the demonstration data has an excessively dense time axis. Even though simply moving toward an object or picking it up and transporting it to a target point would naturally require sufficient speed, if the demonstration data is recorded too densely, the policy moves slowly by mimicking even these simple movement steps in minute detail. This leads to an unnecessary increase in total work time and causes a significant decline in robot productivity in actual industrial applications or application environments requiring high-speed operations.

[0005] Furthermore, methods that uniformly downsample demonstration data or arbitrarily remove frames for the sake of simple speed improvement have the problem of causing serious distribution distortion. Stages requiring fine-tuning, such as precise alignment just before contact or stable object grasping, require the maintenance of dense time-series information; however, if these segments are indiscriminately deleted, the robot loses important manipulation cues. Ultimately, contrary to the intention of increasing execution speed, the uniform downsampling method risks leading to reduced policy stability, increased errors, and higher task failure rates.

[0006] Furthermore, most segment separation methods proposed in existing studies rely on entropy-based analysis or motion statistics-based approaches utilizing clustering, failing to consider the semantic context or spatial context of the demonstration process. Consequently, precision manipulation segments where even the slightest error is unacceptable, such as insertion or alignment tasks, may be incorrectly classified as "segments that can be processed quickly." Conversely, there are cases where segments that can actually be processed quickly, such as simple movement or repetitive transport, are classified as "precision segments," preventing sufficient acceleration. Ultimately, existing methods suffer from a fundamental limitation in that segment classification is unstable because they fail to grasp the semantic characteristics of the demonstration data and the spatial relationships of the manipulation stages.

[0007] Due to these issues, robot policies based on imitation learning fail to utilize the speeds that actual robots can perform and proceed with tasks by inheriting slow speed patterns demonstrated by humans. This results in reduced efficiency and productivity in real-world environments, as well as limitations for tasks requiring high-speed operation. Therefore, a new method is required that can rapidly reconstruct the entire demonstration data while accurately preserving only the precision operation sections. Furthermore, there is a growing need for technology that can accurately distinguish between acceleration-capable and non-acceleration sections by considering both semantic context and spatial characteristics. The problem to be solved

[0008] The present invention was devised to solve the aforementioned problems, and aims to provide a system and method for improving the execution speed of a robot that can improve the execution speed of the robot by accurately distinguishing between precision sections and acceleration-capable sections by considering the semantic and spatial context of demonstration data and efficiently accelerating the entire data.

[0009] However, the technical problems that the present invention aims to solve are not limited to those described above, and other unmentioned problems will be clearly understood by those skilled in the art from the description of the invention below. means of solving the problem

[0010] A system for improving the execution speed of a robot according to one aspect of the present invention comprises: a memory for storing demonstration data of a robot; and by executing instructions stored in said memory, acquiring two-dimensional positions of an object and a gripper from video frames included in the demonstration data; calculating three-dimensional coordinates corresponding to said two-dimensional positions using a monocular depth estimation model; generating relationship information expressing changes in the spatial relationship between the object and the gripper by calculating the temporal rate of change and distance between said calculated three-dimensional coordinates; performing semantic analysis by inputting scene information acquired on a frame-by-frame basis along with said relationship information; classifying each segment of the demonstration data into a precision motion segment or a non-precision motion segment based on the result of said semantic analysis; applying a time-axis alignment algorithm to one of the classified segments as a reference demonstration segment to propagate the segment label of the reference demonstration segment to other demonstration segments of the demonstration data set; and downsampling according to said propagated label by maintaining the original density for the precision motion segment and expanding the frame interval for the non-precision motion segment, wherein the amount of movement of the robot after downsampling is equal to the amount of movement of the robot in said demonstration data and It includes a processor configured to generate training demonstration data adjusted to be identical.

[0011] Preferably, the processor may be configured to calculate the three-dimensional coordinates by back-projecting the two-dimensional positions and monocular depth values ​​of the object and gripper into three-dimensional space corresponding to the intrinsic parameters of the camera.

[0012] Preferably, the processor may be configured to analyze the distance between the calculated three-dimensional coordinates and the temporal change pattern of the distance, and to classify each section of the demonstration data into a precision operation section if a distance reduction trend or minute fluctuation exists.

[0013] Preferably, the processor may be configured to calculate the boundary between the precise operation range and the non-precision operation range by combining the semantic analysis result and the relationship information.

[0014] Preferably, the time axis alignment algorithm may be configured to align a reference demonstration section and a demonstration section other than the reference demonstration section using a time series feature including a motion command, a motion change amount, a joint velocity, a motion curvature, or a combination thereof for the reference demonstration section and the demonstration section other than the reference demonstration section.

[0015] Preferably, the time axis alignment algorithm may be configured to be performed in a manner that limits the alignment path search range by applying a band having a width set in proportion to the sequence length.

[0016] Preferably, the processor may be configured to perform downsampling by forming a set of iterative frames having a plurality of starting points for a non-precision operation section, and then selecting a frame from each set at a certain interval.

[0017] Preferably, the processor may be configured to maintain geometric consistency by adjusting the frame configuration or temporal length so that the cumulative displacement of the robot motion in the downsampled section is the same as the cumulative displacement in the section of the demonstration data.

[0018] Preferably, the processor may be configured to correct the reliability of the segment classification by combining at least one of the robot's joint state information or joint velocity information included in the demonstration data with the relationship information to improve the classification accuracy of the precision motion segment and the non-precision motion segment.

[0019] A method for improving the execution speed of a robot according to one aspect of the present invention comprises: a step of acquiring two-dimensional positions of an object and a gripper from video frames included in demonstration data of the robot, calculating three-dimensional coordinates corresponding to the two-dimensional positions using a monocular depth estimation model, and generating relationship information that expresses changes in the spatial relationship between the object and the gripper by calculating the temporal rate of change and distance between the calculated three-dimensional coordinates; a step of performing semantic analysis by inputting scene information acquired on a frame-by-frame basis along with the relationship information, and classifying each segment of the demonstration data into a precision motion segment or a non-precision motion segment based on the results of the semantic analysis; a step of applying a time-axis alignment algorithm to one of the classified segments to propagate the segment label of the reference demonstration segment to another demonstration segment of the demonstration data set; and a step of downsampling according to the propagated label by maintaining the original density of the precision motion segment and expanding the frame interval of the non-precision motion segment, thereby generating training demonstration data adjusted so that the amount of movement of the robot after downsampling is the same as the amount of movement of the robot in the demonstration data. Effects of the invention

[0020] According to an embodiment of the present invention, by accurately extracting the three-dimensional relationship between the object and the gripper included in the demonstration data and combining it with the results of semantic analysis to stably classify the precision motion segments and the non-precision motion segments that reflect the semantic context of the demonstration data, the robot can effectively accelerate only the segments that actually need to be performed slowly at the original density while maintaining geometric consistency in the segments that do not.

[0021] In addition, if interval labels are generated for only one reference demonstration, the same labels are automatically propagated to other demonstration data through a time-axis alignment algorithm, thereby drastically reducing the labeling cost for the entire set of demonstrations and maintaining consistency in interval classification.

[0022] Furthermore, by applying a downsampling strategy and a cumulative displacement correction technique to the non-precision motion section, the geometric shape of the robot motion trajectory remains the same as the original even after the number of frames is reduced, and the finally generated training demonstration data provides the effect of simultaneously achieving faster operation speed and stability compared to the original demonstration when learning the imitation learning policy.

[0023] In addition, various other additional effects may be achieved by various embodiments of the present invention. These various effects of the present invention are described in detail in each embodiment, or the description of effects that are easily understood by those skilled in the art is omitted. Brief explanation of the drawing

[0024] The following drawings attached to this specification illustrate preferred embodiments of the present invention and serve to further enhance understanding of the technical concept of the present invention together with the detailed description of the invention provided below; therefore, the present invention should not be interpreted as being limited only to the matters described in such drawings. FIG. 1 is a diagram illustrating components for performing a method to improve the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning according to an embodiment of the present invention. FIG. 2 is a diagram illustrating the elements constituting a system for performing a method to improve the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning according to an embodiment of the present invention. Figure 3 is a diagram showing an example of a method to improve the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning implemented by the system of Figure 2. Figure 4 is a block diagram showing the overall configuration of a system that extracts three-dimensional relationship information between an object and a gripper from a demonstration video, divides precise / non-precision segments using VLM and LLM, and then generates an accelerated demonstration data set through segment-by-sampling and band-limited DTW label propagation. Figure 5 is a diagram showing various task example screens demonstrated by applying the system to actual robot operation tasks such as sorting, putting a pen in a cup, conveyor work, and kitchenware operation. Figure 6 is a table showing the results of measuring the success rate and episode length of the method of the present invention compared with ACT and DP-based methods for Insertion and Transfer Cube operations in an ALOHA simulation environment. Figure 7 is a graph showing the different ways in which precision segments are detected by comparing the entropy-based segmentation of DemoSpeedUp and the semantic-distance-based segmentation of the present invention on the time axis through entropy curves, gripper-object distance curves, and example frames. Figure 8 is a table showing the results of comparing the success rate, episode length, and average speed improvement when the ACT and DP-based methods, DemoSpeedUp, and the method of the present invention were applied to various operation tasks in a BiGym simulation environment, respectively. Figure 9 is a table showing the results of comparing the success rate, episode length, and average performance when applying the ACT and DP series methods, DemoSpeedUp, and the method of the present invention to Pen in cup, Sort, Kitchenware, and Conveyor tasks in actual robot experiments. FIG. 10 is a table of ablation experiment results showing the IoU of segment division and the number of detected segments by comparing the method of the present invention with the Insertion and Transfer Cube operations, the case where distance information (r) is removed, the case where scene description information (Description) is removed, and the case where both information is removed. FIG. 11 is a flowchart illustrating a method for improving the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning according to an embodiment of the present invention. Specific details for implementing the invention

[0025] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. Prior to this, terms and words used in this specification and claims should not be interpreted as being limited to their ordinary or dictionary meanings, and should be interpreted in a meaning and concept consistent with the technical spirit of the present invention, based on the principle that the inventor can appropriately define the concept of the terms to best describe his invention.

[0026] Therefore, it should be understood that the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the present invention and do not represent all of the technical ideas of the present invention, and that various equivalents and modifications that can replace them may exist at the time of filing this application.

[0027] FIG. 1 is a diagram illustrating components for performing a method to improve the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning according to an embodiment of the present invention; FIG. 2 is a diagram illustrating elements constituting a system for performing a method to improve the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning according to an embodiment of the present invention; FIG. 3 is a diagram showing an example of a method to improve the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning implemented by the system (100) of FIG. 2; FIG. 4 is a block diagram showing the overall configuration of a system that extracts 3D relationship information between an object and a gripper from a demonstration video, divides precise / non-precision sections using VLM and LLM, and then generates an accelerated demonstration data set through section-by-section downsampling and band-limited DTW label propagation; FIG. 5 is a diagram showing various demonstrations in which the system is applied in actual robot operation tasks such as sorting, putting a pen in a cup, conveyor work, and kitchenware operation. This is a drawing showing an example screen of the work.

[0028] Referring to FIGS. 1 to 5, a robot execution speed enhancement system (100) according to one embodiment of the present invention may include a memory (110) and a processor (120). However, it is not limited thereto, and other general-purpose components may be further included in the robot execution speed enhancement system (100).

[0029] Memory (110) may be configured to store instructions of a computer program or application that implement a method for improving the execution speed of a robot, and a processor (120) may execute the instructions stored in memory (110) to execute the program or application. For example, memory (110) may be implemented as non-volatile memory such as ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, etc., or volatile memory such as DRAM, SRAM, SDRAM, PRAM, RRAM, FeRAM, etc., and may be implemented in the form of HDD, SSD, SD, Micro-SD, etc., or a combination thereof. The processor (120) may be implemented as an array of logic gates, a microprocessor, a CPU, a GPU, an AP, or a combination thereof.

[0030] Additionally, the robot execution speed enhancement system (100) includes an AI model and can perform a method to enhance the robot's execution speed through the learning of the AI ​​model. The AI ​​model may be learned based on a supervised learning method, but is not limited thereto, and may also be learned based on unsupervised learning or semi-persistent learning methods. Additionally, the AI ​​model may be composed of one or more neural network layers, and each neural network may include one or more weights. The AI ​​model may perform learning / inference by performing operations between one or more weights and input data. In one embodiment, the AI ​​model is a Generative AI model, which is an artificial intelligence system that generates new content or information based on input data, and may also include an AI model specialized in Transformer-based Natural Language Processing (NLP), but is not limited thereto.

[0031] The present invention relates to a technology that enables a robot to systematically accelerate unnecessarily slow sections while maintaining the precision required for parts of the demonstration data during the process of performing manipulation tasks by mimicking human demonstration data. To this end, the present invention is configured to improve the execution speed of a robot by interpreting the spatial relationship between objects and grippers included in the demonstration data in a three-dimensional coordinate system, classifying the demonstration section into precision and non-precision sections according to the semantic context of the scene, and adjusting the temporal density of the entire demonstration data using the classification results. This process consists of demonstration video analysis, semantic-based section classification, label propagation through time-axis alignment, and data acceleration through frame reconstruction, as illustrated in FIG. 4.

[0032] Specifically, the memory (110) stores demonstration data of the robot, and this demonstration data may include video frames, time information, depth estimation results, and robot behavior information. Since demonstration data is generally performed slowly in many cases to ensure safety and accuracy in the operation phase, raw demonstration data has a problem in that the intervals between frames are very close and the time is excessively long. This slow pattern becomes a factor that limits the actual execution speed of the robot if the imitation learning policy learns it as is.

[0033] In one embodiment, the memory (110) can store a demonstration data set consisting of a plurality of demonstration data.

[0034] Then, the processor (120) can first obtain the two-dimensional positions of the object and the gripper from the video frames included in the demonstration data. At this time, the two-dimensional coordinates of the object center point and the gripper center point for each frame are calculated using an object tracking model such as Grounded-SAM2. Subsequently, the processor (120) obtains a depth value corresponding to each pixel using a monocular depth estimation model, and the obtained two-dimensional coordinates (u, v) and the depth value Z are back-projected into three-dimensional coordinates using camera intrinsic parameters. This process can be expressed by the following mathematical formula 1.

[0035] [Mathematical Formula 1]

[0036]

[0037] The result of this back-projection represents the actual 3D position of the object and the gripper, and serves as a basis for calculating changes over time.

[0038] Additionally, the processor (120) can generate relationship information that represents the change in spatial relationship between the object and the gripper by calculating the temporal rate of change and distance between the three-dimensional coordinates calculated above.

[0039] Specifically, the processor (120) uses these three-dimensional coordinates to calculate the Euclidean distance between the object and the gripper, and the object-gripper distance at frame t. is defined by the following mathematical formula 2.

[0040] [Mathematical Formula 2]

[0041]

[0042] Since this distance value reflects semantic behavioral changes such as approach, contact, and retraction occurring during the manipulation phase, it serves as a key clue for distinguishing the demonstration segment into precision and non-precision motion segments. For example The section where the value decreases is likely the precision stage where the gripper approaches the object, and Sections where the value increases or changes gradually are likely to be non-precision steps, such as transfer or intermediate movements.

[0043] Additionally, the processor (120) can perform semantic analysis by inputting scene information acquired in frame units along with relationship information, and classify each section of the demonstration data into a precise operation section or a non-precision operation section based on the results of the semantic analysis.

[0044] Specifically, the processor (120) performs semantic analysis by combining this distance-based relationship information as well as scene information acquired on a frame-by-frame basis. Semantic analysis generates a description of the scene using a VLM (such as an InternVL) and interprets the generated sentence information through an LLM to infer what task the robot is performing, whether a specific section is a dangerous stage immediately before contact, or a simple movement stage. Unlike conventional entropy-based segmentation, as can be seen in the “Scene Description” and “Semantics-based Segmentation” sections of FIG. 4, this semantic-based judgment enables accurate identification of precise sections by simultaneously reflecting the semantic structure of the scene and changes in three-dimensional relationships.

[0045] When semantic analysis and relationship information are combined, the processor (120) can classify each segment of the demonstration data into a precise operation segment or a non-precision operation segment. This classification is much more precise because it reflects high-level semantic information such as the type of object, the purpose of operation, and the relationship with surrounding objects, rather than simply separating based on changes in distance, and this is visualized as the segments being divided into yellow / purple colors on the top timeline of FIG. 4.

[0046] Additionally, the processor (120) can apply a time axis alignment algorithm to one of the classified intervals to propagate the interval label of the reference interval to another interval of the demonstration data set. This performs frame-to-frame matching using banded dynamic time alignment (banded DTW) and is used to align two sequences of different lengths without distortion. The frame characteristics used at this time consist of robot motion commands, behavior change amounts, joint velocities, etc., and these feature vectors are defined as shown in Equation 3 below.

[0047] [Mathematical Formula 3]

[0048]

[0049] Such dynamics-only features are not affected by LLM or VLM and enable alignment based purely on geometric and dynamic information. Band-limited DTW prevents alignment between frames that are too far apart, thereby ensuring the accuracy and efficiency of alignment calculations.

[0050] Once alignment is complete, precision / casual zone labels generated in the reference episode are propagated to other episodes to ensure they follow the same temporal pattern. This process significantly reduces the manual burden of labeling the entire demonstration data.

[0051] Additionally, the processor (120) may be configured to generate training demonstration data adjusted so that the amount of movement of the robot after downsampling is the same as the amount of movement of the robot in the demonstration data, by downsampling the precise movement section according to the propagated label by maintaining the original density and expanding the frame interval of the non-precision movement section.

[0052] Specifically, when label propagation is complete, the processor (120) performs downsampling based on the propagated label by maintaining the original density in the precision operation section and expanding the frame interval in the non-precision operation section. Downsampling in the non-precision section is performed according to the replicate-before-downsample (RBD) strategy, which is expressed by the following mathematical formula 4.

[0053] [Mathematical Formula 4]

[0054]

[0055] Here is a demonstration sequence from interval si to ei, and RBD represents the process of constructing a repeating sequence with multiple starting points and sampling at regular intervals. This allows for reducing unnecessarily dense frames in non-precision intervals while maintaining the structural form of the manipulation trajectory.

[0056] However, simply reducing the number of frames results in a difference in the total displacement of the trajectory between the original demonstration and the downsampled demonstration, which can cause the imitation learning policy to learn an abnormal path. To prevent this, the present invention applies a geometric consistency condition represented by the following mathematical formula 5.

[0057] [Mathematical Formula 5]

[0058]

[0059] Here represents the amount of state change between frames, signifying the position change vector of the robot end effector or the change vector in joint space. The above equation implies that the frame composition or sequence length must be adjusted to maintain the same level as the cumulative movement distance of the original demonstration even after downsampling, and in the present invention, the entire sequence is corrected to satisfy this. As a result, the downsampled sequence becomes shorter in time while maintaining the geometric structure of the original demonstration, and this result is used as “training demonstration data” to train an imitation learning policy.

[0060] Figure 5 illustrates an example in which training demonstration data generated by applying the present invention is applied to actual robot tasks (Sort, Pen in cup, Conveyor, Kitchenware). A policy trained using training demonstration data generated through the method of the present invention shows results in eliminating time waste in non-precision sections while maintaining or improving the success rate in precision sections. This clearly distinguishes the present invention from existing technologies in that it does not simply reduce demonstration data, but preserves only the necessary parts by reflecting semantic and geometric contexts.

[0061] According to this embodiment of the present invention, the three-dimensional relationship between the object and the gripper included in the demonstration data is accurately extracted and combined with the results of semantic analysis to stably classify precision motion sections and non-precision motion sections that reflect the semantic context of the demonstration data, thereby allowing the robot to effectively accelerate only the sections that actually need to be performed slowly at the original density, while maintaining geometric consistency in the sections that do not.

[0062] In addition, if interval labels are generated for only one reference demonstration, the same labels are automatically propagated to other demonstration data through a time-axis alignment algorithm, thereby drastically reducing the labeling cost for the entire set of demonstrations and maintaining consistency in interval classification.

[0063] Furthermore, by applying a downsampling strategy and a cumulative displacement correction technique to the non-precision motion section, the geometric shape of the robot motion trajectory remains the same as the original even after the number of frames is reduced, and the finally generated training demonstration data provides the effect of simultaneously achieving faster operation speed and stability compared to the original demonstration when learning the imitation learning policy.

[0064] In one embodiment, the processor (120) may be configured to calculate the three-dimensional coordinates by back-projecting the two-dimensional positions and monocular depth values ​​of the object and gripper into three-dimensional space corresponding to the intrinsic parameters of the camera.

[0065] This process consists of the step of extracting the positions of the object and the gripper from the demonstration video as illustrated in FIG. 4, and combining them with the depth estimation results to reconstruct their positions in actual 3D space. First, pixel coordinates (u, v) corresponding to the object and the gripper in each frame are calculated from the video frames included in the demonstration data using an object and gripper detector such as the Grounded-SAM2 model. Since accurate tracking may be difficult with such 2D position information because the object may be partially obscured or shadows may appear during repeated manipulation processes, this embodiment applies SAM2-based mask tracking and bounding box linkage in combination to ensure that the positions of the object and the gripper are stably maintained. As a result, the ID for the same object can be consistently maintained across the frames.

[0066] In addition, the processor (120) calculates the depth value Z of each frame using a monocular depth estimation model (Video Depth Anything, DA2, etc.). The monocular depth estimation model can estimate the relative depth within a scene using only a single image without a stereo camera or LiDAR, and can use the relative depth value scaled in meters or other units. This depth value is an essential element for determining the distance difference between an object and a gripper within the image, and is particularly important information for correctly determining changes in the timing of the gripper approaching an object or minute movements just before contact during robot manipulation operations.

[0067] The 2D coordinates (u, v) and depth value Z obtained in this way are used together with the camera's intrinsic parameters (a matrix denoted by K) to be converted into 3D coordinates. This conversion process is performed using a standard back-projection formula, which is expressed by the equation corresponding to Equation 1 mentioned above.

[0068] In mathematical formula 1 is the inverse matrix of the camera intrinsic parameters, and is a vector that normalizes the image coordinate system. The above equation serves to restore the 3D position of a point by projecting it onto the camera coordinate system based on the depth value Z. In other words, when the pixel position occupied by an object or gripper in the image is combined with the depth value of that pixel, it is possible to accurately calculate where that point exists in the actual physical space.

[0069] This back-projection process serves as the basis for accurately interpreting the spatial meaning of demonstration data and is an essential step for quantifying changes in distance between an object and a gripper. For example, if the x, y, and z coordinates of the object and the gripper are calculated for each frame, the robot can track movements such as approaching or retreating from an object during a specific task, which serves as a basis for reliably distinguishing precision motion segments in subsequent steps. Furthermore, to reduce distortion or lens distortion issues inherent to image-based information, a correction process using intrinsic parameters is performed beforehand, ensuring that all 3D coordinates are consistently represented based on the same camera coordinate system.

[0070] As such, the processor (120) of this embodiment can calculate a 3D position in the camera coordinate system by combining the 2D coordinates of the object and the gripper with the depth value obtained from monocular depth estimation, and these calculated 3D coordinates become essential input values ​​that enable the calculation of the Euclidean distance between the object and the gripper, the analysis of the rate of change of time, and semantic-based segment classification in subsequent steps of the present invention. Therefore, the 3D back-projection process in this embodiment plays a key technical role in distinguishing between precise operation stages and non-precision operation stages as intended by the entire system, and subsequently performing segment-by-sampling and improving execution speed.

[0071] In one embodiment, the processor (120) may be configured to analyze the distance between the calculated three-dimensional coordinates and the temporal change pattern of the distance, and to classify each section of the demonstration data into a precision operation section if there is a distance decrease trend or a fine fluctuation.

[0072] Additionally, the processor (120) may be configured to calculate the boundary between the precise operation section and the non-precision operation section by combining the semantic analysis results and relationship information.

[0073] In this embodiment, the 3D coordinates used by the processor (120) are values ​​back-projected into the camera coordinate system using the 2D positions (u, v) and depth values ​​(Z) of the object and gripper from the previous step, and the distance value between the object and the gripper based on these coordinates It is calculated as in the aforementioned mathematical formula 2.

[0074] Since the temporal change in this distance value reflects whether the robot is approaching an object or moving after grasping the object, the processor (120) analyzes the pattern of distance reduction, stagnation, and fine vibration to determine whether the corresponding section is a precision motion stage directly related to the success of the task. For example, in the section where the gripper converges to the center point of the object It continuously decreases or exhibits minute vibrations, indicating that this is a stage requiring very precise control during the actual manipulation process. On the other hand, in the stages of moving an object or switching between work areas after grasping it, Since the range of change of becomes large or remains constant, these sections are considered to be non-precision sections.

[0075] However, since it is difficult to fully understand the entire work context based solely on distance information, and there are cases where the distance is calculated unstably due to visual factors such as lighting changes or occlusion, the present embodiment combines the semantic analysis results with distance-based relationship information to calculate the segment boundaries more stably. The processor (120) generates a scene description for each frame and inputs it into the LLM to derive a semantic label indicating the work intention.

[0076] For example, in cases where it is determined that object approach or precise manipulation is required, such as “approaching target” or “aligning tool for insertion,” within the same frame If it decreases or the displacement fluctuates very slightly, that section is determined to be a precision operation section. Conversely, descriptions such as “carrying object” or “moving to next zone” are generated. If it changes significantly or remains constant, the corresponding section is classified as a non-precision operation stage.

[0077] Furthermore, by incorporating stability rules—such as minimum interval length (L_min), interval merging criteria (G_min), and hysteresis rules—it ensures that momentary noise or frame-level misclassification does not affect the overall interval labels. Moreover, for frames where the semantic model exhibits low confidence, a coverage completion step is performed to fill the gaps by extending labels from adjacent high-confidence intervals. This process ensures that the boundary between precise and imprecise intervals remains continuous and semantically consistent, thereby enabling stable performance of time-axis alignment (DTW) and downsampling in subsequent steps.

[0078] The result of such semantic-based interval partitioning is defined in the preceding mathematical equation 4 It is expressed as a set of intervals of the form.

[0079] Each segment calculated through semantic analysis and distance pattern analysis in the above equation Different downsampling scales N_{yi} are applied depending on whether yi is a precision interval or a causal interval. That is, the precision interval is maintained close to N_precision = 1 to preserve the original density, and the causal interval is sampled at larger intervals of N_casual > 1, and this process is performed by the replicate-before-downsample (RBD) strategy. The processor (120) selectively adjusts the frame interval according to the interval-specific label yi so that all sequences of the entire demonstration dataset D are reconstructed with reasonable temporal density.

[0080] This mathematical formula 4 is the “section boundary” calculated by the present embodiment It clearly demonstrates that ” functions as a key input value that goes beyond simple classification and subsequently transforms the entire demonstration data set. That is, the distance-semantic-based boundary calculation process performed by the processor (120) in this embodiment is directly connected to the subsequent RBD downsampling and time alignment, and ultimately serves as a basis for transforming the entire demonstration data set into an accelerated form of T(D,S,N).

[0081] Consequently, according to the present embodiment, the processor (120) can derive semantically stable precise / non-precision interval boundaries by combining distance-based relationship information and semantic analysis results, and since these boundaries are used as essential inputs for adjusting the time axis density of the entire demonstration data set by interval according to Equation 4, a structure is achieved in which the robot can stably obtain an effect of improving the actual execution speed.

[0082] In one embodiment, the time axis alignment algorithm may be configured to align a reference demonstration section and a different demonstration section using a time series feature including a motion command, a motion change amount, a joint velocity, a motion curvature, or a combination thereof for a reference demonstration section and a different demonstration section.

[0083] Additionally, the time axis alignment algorithm can be configured to limit the alignment path search range by applying a band having a width set in proportion to the sequence length.

[0084] In this embodiment, the processor (120) performs time-axis alignment by applying a Banded Dynamic Time Warping (DTW) algorithm to extend the labels of the precision motion section and the non-precision motion section derived from the reference demonstration section to other demonstration sections of the entire demonstration data set. This algorithm is configured to align the structural patterns of the actual motion performed by the robot stably and efficiently by utilizing proprioception-based features rather than image-based features.

[0085] To this end, the processor (120) for each frame t has a time series feature as shown in the aforementioned mathematical formula 3 Constitutes. is an operation command , change in command between adjacent frames , joint speed , change in velocity , size of the command vector , magnitude of the velocity vector , magnitude of the command change , magnitude of the change in velocity , amount of change in joint position , and the angle representing the rate of change of direction of the command vector and the velocity vector , Features configured in this way are all normalized using the Z-score method, allowing comparison to be performed based on the change patterns of each sequence without being affected by differences in absolute numerical magnitudes during DTW alignment. This feature configuration can reflect even minute structural changes in robot motion, enabling precise measurement of the degree of similarity between the actual operation patterns of two sequences without image-based information.

[0086] The processor (120) has a reference sequence Lo, another sequence It is set to and performs DTW alignment based on the similarity between the two sequences. However, traditional DTW is Since the computational load is very large due to searching the entire alignment space, in this embodiment, the width is set in proportion to the sequence length Limit the search bandwidth by applying. Here is set to a value in the range of approximately 0.05 to 0.10. Such band limiting induces the DTW to find the alignment path only in a reasonable search space near the diagonal in the form of a Sakoe-Chiba structure, thereby reducing unnecessary searches and significantly improving computational efficiency. In particular, this enables rapid alignment to be performed within a quasi-linear level of computation time, even in robot demonstration data with lengths ranging from hundreds to thousands of frames.

[0087] The processor (120) performs this band-limited DTW to calculate an optimal path P that minimizes the cumulative cost, and then this path is converted into a monotonically increasing mapping so that the time order of the original sequence is maintained. That is, from frame 1 of the reference sequence Mapping m is configured so that it corresponds to frames of another sequence in a monotonically increasing manner. In the DTW alignment result, multiple target frames may correspond to the same reference frame or there may be decreasing sections; however, in this embodiment, to eliminate such non-linearity, each corresponding frame is averaged or the order is rearranged so that a mapping in the form of a monotonically increasing form that does not decrease is finally obtained. Through this process, the precise / non-precision labels assigned in the reference demonstration section can be naturally transferred to other demonstration data without order distortion.

[0088] Meanwhile, since the label boundaries may be inaccurately matched due to differences in frame counts, noise, or local distortion of the DTW if the coarse boundary values ​​obtained by DTW alignment are used as is, a boundary refinement technique is additionally applied in this embodiment. Specifically, the reference demonstration section The coarse interval corresponding to After obtaining the values, the boundaries are corrected by snapping the endpoints of the two intervals to the point where the ℓ₂ distance is minimized within ±W frames, respectively. W is typically set to 12 frames, and within this window, the point where the feature characteristics of the reference sequence and the target sequence best match is found. Through this, the semantic boundaries of the precision operation interval are corrected so that they remain the same in other sequences.

[0089] If an overlap occurs between several aligned segments, the processor (120) removes it to organize the aligned segments while maintaining the segment order, and if there are unmatched frames, it applies a keep label by default in limited situations to prevent semantic gaps from occurring. As a result, the entire sequence has consistent labels without interruption, which is used as an important basis for subsequent downsampling and geometric consistency maintenance processes.

[0090] Consequently, the time-axis alignment algorithm performed by the processor (120) in this embodiment enables the precise / non-precision segment labels of a reference demonstration to be reliably and efficiently propagated to other demonstrations through a DTW-based alignment method that combines the multidimensional feature configuration presented in Equation 3 and a band limit having a width proportional to the sequence length. This method ensures computational efficiency while guaranteeing the reliability of the label propagation process by accurately reflecting the structural similarity of the actual robot motion pattern.

[0091] In one embodiment, the processor (120) may be configured to perform downsampling by configuring a set of repeating frames having a plurality of starting points for a non-precision operation section, and then selecting a frame at a certain interval from each set.

[0092] Additionally, the processor (120) may be configured to maintain geometric consistency by adjusting the frame configuration or time length so that the cumulative displacement of the robot motion in the downsampled section is the same as the cumulative displacement in the section of the demonstration data.

[0093] In this embodiment, the processor (120) first identifies a section classified as a non-precision movement through semantic and relational-based analysis. This non-precision movement section corresponds to a stage where the robot does not handle an object precisely, but rather, for example, moves to a destination after grasping an object or performs simple transport, and in such sections, there is a relatively low need to maintain temporal tightness. However, downsampling by simply skipping frames at regular intervals drastically damages the visited-state diversity present in the original demonstration, causing the problem of errors accumulating during imitation learning.

[0094] To address this, the processor (120) of the present embodiment processes non-precision intervals according to a replicate-before-downsample (RBD) strategy. This strategy generates N replicas with different phases from a single sequence, and then selects a frame from each replica according to the same rule to downsample it; it is distinctly different from conventional uniform interval sampling in that it can preserve the original state distribution even if the sequence length is reduced. For example, a sequence interval When N replicas are generated for , the first replica starts at s_i, the second replica starts at s_i+1, and the last replica starts at s_i+N-1, thereby observing the same interval at different relative starting points. Then, the processor (120) selects frames by skipping each replica at a constant interval N to generate a downsampled sequence. This process is identical to that expressed by the aforementioned Equation 4, which indicates that the present embodiment performs RBD transformation for non-precision intervals and combines these transformed sequences to form a final accelerated demonstration dataset.

[0095] An important point in Equation 4 is that N_{y_i} varies depending on whether y_i is precise or non-precision; while p-precision intervals undergo almost no downsampling to maintain the original density, casual (non-precision) intervals perform temporal compression by setting a large value for N. In other words, Equation 4 explicitly demonstrates that the RBD step of this embodiment is not a simple frame reduction, but rather dynamic downsampling performed selectively based on the interval label y_i.

[0096] However, as temporal density decreases due to RBD, the interval between frames widens, so there is a possibility that the downsampled trajectory may not match the actual travel distance of the original trajectory. For example, if the robot's end-effector travels 20 cm in the original section, the downsampled trajectory must also travel the same 20 cm to preserve spatial progression during the imitation learning process. If the trajectory travels only 10 cm after downsampling, the learned policy will learn a biased behavior that performs a much smaller amount of movement than the actual demonstration.

[0097] In this embodiment, the processor (120) corrects the horizontal length K′ so that the cumulative displacement in the downsampled section is equal to the cumulative displacement in the original section in order to maintain this geometric consistency. The aforementioned Equation 5 represents this process.

[0098] In mathematical formula 5 represents the end-effector pose or low-dimensional state representation. The left-hand term of Equation 5 is the cumulative displacement in the horizontal K′ adjusted after downsampling, and the right-hand term is the cumulative displacement in the original horizontal K.

[0099] By adjusting these two values ​​to be as identical as possible, the temporal compression caused by downsampling is prevented from causing spatial distortion.

[0100] The processor (120) can perform corrections in various ways for this purpose. For example, if the gap between frames in the downsampled trajectory is too wide and the cumulative displacement is insufficient, K′ can be increased to include more frames, and conversely, if the cumulative displacement becomes excessively large, K′ can be reduced to correct the overall length of the trajectory. In addition, the frame itself selected during the RBD process can be finely adjusted to achieve a distributed balance, all of which are measures to satisfy the equivalent displacement condition of Equation 5.

[0101] When this geometric consistency correction is performed, the downsampled sequence maintains a spatial movement trajectory with the same structure as the original sequence, even though it is shortened in time. As a result, during the imitation learning process, the robot policy can learn demonstrations that have reduced actual time but the same spatial meaning, which enables policy learning capable of performing stable actions while maintaining fast manipulation speeds. Experimentally, ESPADA achieves both a higher success rate and faster policy execution speeds compared to conventional simple downsampling through this geometric consistency correction.

[0102] In conclusion, the RBD-based downsampling of the non-precision section performed by the processor (120) in this embodiment and the geometric consistency correction according to Equation 5 perform the function of efficiently reducing the temporal density of the demonstration data while faithfully preserving the shape of the original spatial trajectory. This is a key element that enables the subsequent imitation learning policy to stably learn high-speed motion based on undistorted state information, and the advantage is achieved that the entire demonstration dataset takes on a form that is significantly accelerated compared to the original, while the structural and spatial elements for successful robot operation remain intact.

[0103] Figure 6 is a table showing the results of measuring the success rate and episode length of the method of the present invention compared with ACT and DP-based methods for Insertion and Transfer Cube operations in an ALOHA simulation environment.

[0104] Specifically, Figure 6 is a table showing the results of comparing the existing method (DemoSpeedup), a simple 2x speed playback technique, and the method of the present invention for Insertion and Transfer Cube tasks in an ALOHA simulation environment, for ACT-based policies and Diffusion Policy (DP)-based policies, respectively. As shown in the figure, the method of the present invention demonstrates the effect of significantly shortening episode lengths while maintaining or even improving the success rate in both representative operation tasks. For example, in the case of Insertion tasks, compared to the existing ACT, the method of the present invention increases the success rate from 21% to 36% and shortens the episode length from 452 frames to 107 frames by approximately 76%. Similarly, in Transfer Cube tasks, it can be seen that the episode length is shortened from 291 frames to 72 frames by approximately 75% without slightly lowering the success rate from 72% to 38%. This contrasts with ACT-2x (success rate 13%·238 frames) or DP-2x (success rate 12%·245 frames), which simply increase speed and significantly reduce the success rate, and clearly demonstrates that the downsampling method of the present invention provides a stable acceleration effect that improves execution speed without reducing the success rate.

[0105] In particular, in the Insertion task of Figure 6, the success rate of the existing entropy-based acceleration technique (DemoSpeedup) is 32% in the ACT structure and 14% in the DP structure, whereas the method of the present invention achieves 36% and 10%, respectively, achieving a success rate equivalent to or exceeding that of the entropy-based method, while maintaining an episode length that is similar to or shorter than that of DemoSpeedup. This is because the instability of the initial approaching phase, which frequently occurs in the ALOHA environment, causes misclassification in the entropy-based technique, incorrectly labeling the corresponding section as "casual," whereas the method of the present invention can accurately identify the corresponding section as a precision section by combining the trend of 3D distance change between the gripper and the object with semantic information. As the stability of such section classification is a key factor determining model performance, it is shown that the IoU of the Insertion task is improved to 0.1989 in the semantic-based method identical to the present invention compared to 0.1745 in the entropy-based method.

[0106] In addition, as shown in Fig. 6, the method of the present invention demonstrates a more stable success rate compared to DemoSpeedup in the Transfer Cube task, thanks to segment classification that reflects the semantic context of the task and the spatial interaction between the object and the gripper, rather than just the level of simple entropy. Entropy-based methods have errors in that they incorrectly identify periodic transition movements as “precision segments” to retain excessive samples, while conversely classifying initial unstable segments as accelerateable. However, the method of the present invention prevents these errors by utilizing semantic parsing and distance change signals.

[0107] Accordingly, FIG. 6 clearly demonstrates the effect of the system of the present invention propagating safe acceleration labels across the entire dataset based on semantic and spatial interval information of a single demonstration, and subsequently selectively improving only the operation speed while maintaining the original operation success rate through the replicate-before-downsample and geometric consistency adjustment processes. In other words, by realizing safe acceleration that preserves the success rate rather than simply reducing time, the present invention provides a technical effect that overcomes the limited speed characteristics of human demonstrations and significantly improves the practical usability of imitation learning policies.

[0108] Figure 7 is a graph showing the different ways in which precision segments are detected by comparing the entropy-based segmentation of DemoSpeedUp and the semantic-distance-based segmentation of the present invention on the time axis through entropy curves, gripper-object distance curves, and example frames.

[0109] Figure 7 is a graph comparing the entropy-based segmentation results of the conventional DemoSpeedUp method and the semantic-distance-based segmentation results of the present invention on the time axis, visually demonstrating that the two methods divide the same demonstration section in completely different ways. In the figure, the Entropy Energy curve represents the entropy-based energy signal used by DemoSpeedUp, while the other curve represents the distance between the gripper and the object used in the present invention. It shows the temporal change trend of. In addition, actual robot video frames from the corresponding time period are displayed at the bottom, allowing for an intuitive verification of the semantic transitions of the task (approaching, alignment, manipulation) and the differences in judgment between the two segmentation methods.

[0110] As shown in Figure 7, the DemoSpeedUp method is sensitive to points where entropy values ​​suddenly decrease or increase, and tends to mistake these noise-based signals for the boundaries of precision sections. Since entropy signals are significantly affected by visual factors such as changes in camera viewpoint, lighting reflections, and background textures, rapid fluctuations can occur even in frames unrelated to the semantic transition of actual robot operation. Due to these characteristics, DemoSpeedUp fails to accurately capture the key precision section where the motion transitions from fast to slow in the example section; instead, it incorrectly identifies other sections where the entropy signal is temporarily low as the precision manipulation stage. The range indicated by the gray arrow in the figure is the area classified as the precision section by DemoSpeedUp; however, when compared with the actual operation frame (bottom image), it can be seen that this is the fast movement stage before the robot begins interacting with an object, and is far removed from the precision manipulation stage directly related to the success of the task.

[0111] In contrast, the method of the present invention does not rely on image-based noise such as entropy, but rather considers semantic analysis results and distance change signals together to reliably capture the point at which the manipulation meaning actually changes. In particular, the gripper-object distance curve used in the present invention exhibits a pattern in which it shows a continuous decreasing trend during the approaching-alignment-manipulation process and then rapidly stabilizes just before accurately grasping the object; this indicates that the corresponding section is a stage where precision operation is actually required. The red segmentation result in Fig. 7 combines these distance-based semantic signals with LLM-based semantic judgment to accurately identify the actual precision section that DemoSpeedUp missed. That is, the slow-precision section, from the point indicated by the red dotted line where the robot reduces speed and performs precise movements, is correctly classified as precision operation in the method of the present invention.

[0112] Furthermore, the present invention introduces stability rules and hint-based criteria during the semantic analysis process to perform procedures such as guaranteeing minimum segment length, hysteresis-based transition rules, ignoring minute vibration segments, and correcting semantic continuity. These procedures prevent the problem of segment labels changing frequently due to frames with low confidence in the semantic model or minute fluctuations in distance signals, and ultimately produce semantically consistent precise and non-precision segments. Consequently, the present invention constructs segmentation boundaries that are much more robust and stable compared to DemoSpeedUp, and establishes a foundation for the stable application of subsequent RBD-based downsampling and geometric consistency correction.

[0113] The results shown in Figure 7 suggest that beyond simple segmentation differences, the reliability of segment boundaries has a decisive influence on the learning performance of the imitation learning policy. Unlike the DemoSpeedUp method, the present invention fundamentally enhances the temporal structure of the learning data by accurately designating segments where interaction with real objects is central as precise segments, and designating irrelevant segments that move quickly as casual segments so that they can be boldly accelerated.

[0114] Figure 8 is a table showing the results of comparing the success rate, episode length, and average speed improvement when the ACT and DP-based methods, DemoSpeedUp, and the method of the present invention were applied to various operation tasks in a BiGym simulation environment, respectively.

[0115] Specifically, Figure 8 is a table showing the results of comparing the success rate, episode length, and average speed improvement when applying the DemoSpeedUp method and the method of the present invention to various manipulation tasks (Sandwich Remove, More Plate, Load Cups, Put Cups, Saucepan to Hob, Drawers Close, Cupboard Open) performed in a BiGym simulation environment, respectively, for ACT-based policies and Diffusion Policy (DP)-based policies. As can be seen in the figure, the method of the present invention not only achieves a higher success rate than existing ACT or DP-only policies in various tasks, but also shows a more balanced improvement effect in both success rate and speed when compared to the DemoSpeedUp method. In particular, the ACT-based policy recorded a success rate of 80% in the Sandwich Remove task, showing a higher success rate than the 77% of the existing DemoSpeedUp method, and the episode length also maintained a reduction effect of almost the same level as DemoSpeedUp at 176 frames. In the More Plate task, ACT+DemoSpeedUp shows a success rate of 53%, whereas ACT+Ours lowers the success rate to 24% while maintaining the same 91-frame episode length; however, this reflects the analysis that the task has a very complex semantic structure and that the entropy-based segmentation method has the problem of generating inaccurate labels. Importantly, in most tasks, the present invention is more stable in segment classification than DemoSpeedUp, so there is less variability in the success rate and it achieves higher performance in specific tasks.

[0116] In addition, the present invention provides stable improvement effects compared to DemoSpeedUp in Load Cups and Put Cups tasks. For the Put Cups task, the ACT-based method maintains the success rate at nearly the same level of 62% to 60% while the episode length increases only slightly from 132 frames to 149 frames, whereas the DP-based method shows a significant improvement in the success rate from 21% to 38%. This demonstrates that because precise and non-precision sections are distinguished more accurately based on semantic grounds in the present invention, compounding errors caused by DemoSpeedUp accelerating incorrect sections are reduced, and consequently, the method of the present invention, which selectively accelerates only non-precision sections, provides superior stability.

[0117] The present invention exhibits the same trend in the Saucepan to Hob, Drawers Close, and Cupboard Open tasks presented below. In particular, for the Drawers Close task, ACT+Ours achieves a 100% success rate with a very short episode length of 56 frames; this demonstrates performance that maintains the same success rate while performing operations at a faster speed than DemoSpeedUp (63 frames, 100%). In the Cupboard Open task as well, ACT+Ours maintains a 100% success rate with an episode length of 81 frames, and even in DP-based operations, demonstrates a higher success rate (59%) and a shorter episode length (103 frames) compared to DemoSpeedUp.

[0118] As shown in the average speed improvement figures in the far right column of the table, the present invention achieves a speed improvement of 2.3× based on ACT and 2.1× based on DP, recording higher overall performance compared to DemoSpeedUp's 2.1× and 1.9×. These results provide a balanced acceleration dataset that can simultaneously achieve two goals of speed improvement and success rate preservation by stably detecting precision operation sections based on meaning and distance and selectively accelerating only non-precision sections.

[0119] The results of Figure 8 clearly show that the method of the present invention does not merely reduce demonstration time, but generates safely accelerated demonstrations while maintaining semantic consistency and geometric reliability, and provides the effect of substantially improving the overall performance of the imitation learning policy even in various difficult manipulation tasks.

[0120] Figure 9 is a table showing the results of comparing the success rate, episode length, and average performance when applying the ACT and DP series methods, DemoSpeedUp, and the method of the present invention to Pen in cup, Sort, Kitchenware, and Conveyor tasks in actual robot experiments.

[0121] Specifically, FIG. 9 is a table comparing the success rate, episode length, and average degree of performance improvement when the DemoSpeedUp method and the method according to the present invention are applied to ACT-based policies and Diffusion Policy-based policies, respectively, for Pen in cup, Sort, Kitchenware, and Conveyor tasks performed in an actual robot environment. As can be seen in the figure, the method of the present invention effectively overcomes the typical performance degradation problems experienced by existing methods in the complex and highly uncertain environment of actual robot operation, and demonstrates distinct superiority in that it simultaneously secures fast operation speed and a high success rate.

[0122] First, looking at the Pen in cup task, ACT-based policies show high success rates for ACT alone, DemoSpeedUp, and the method of the present invention; however, in terms of episode length, the method of the present invention completes the task in the shortest time while maintaining the same success rate. For example, although the ACT+DemoSpeedUp method reduces the episode length from 15.52 to 5.36 and may cause issues with semantic and spatial consistency being broken in the process, the method of the present invention maintains the same success rate while achieving a short episode length of 6.57 by more stably distinguishing sections and reflecting distance-based semantic information compared to DemoSpeedUp. This demonstrates that while the DemoSpeedUp method is overly sensitive to changes in entropy signals and attempts to accelerate sections unrelated to the point where the manipulation meaning changes, the method of the present invention can stably achieve a time reduction effect without a decrease in success rate because it comprehensively considers the actual spatial access relationship and semantic meaning between the gripper and the object.

[0123] These differences are more clearly evident in sort tasks. In the ACT-based approach, DemoSpeedUp fails to properly preserve segments requiring precise manipulation, resulting in a slight decrease in success rate to 24 / 30. While episode length decreases, stability is lacking. In contrast, the method of the present invention maintains a success rate of up to 28 / 30 and achieves an episode length of 15.56, similar to DemoSpeedUp. This is achieved because the semantic and distance-based segmentation of the present invention captures minute manipulation steps occurring during the task to accurately preserve precise segments and selectively accelerates only unnecessary movement steps. While simple entropy-based methods are heavily influenced by visual changes and misidentify even semantically unimportant segments as precise movements, the present invention performs stable segment classification that reflects the actual task meaning based on object relationship information and LLM-based scene description.

[0124] The effectiveness of the present invention is also clearly observed in Kitchenware operations. While DemoSpeedUp fails to distinguish between non-precision and precision operations at most points and accelerates excessively to the point where manipulation is required, resulting in an extremely low success rate of 1 / 20, the present invention secures a high success rate of 16 / 20 by accurately recognizing a pattern where the distance between the gripper and the object is slightly reduced and maintained as the precision operation section. This difference demonstrates that the method of the present invention operates much more stably in the nearness manipulation section, which is a weak area for DemoSpeedUp.

[0125] The Conveyor operation, which is the most challenging task, also exhibits the same trend. In Conveyor operations, the target object moves, and there is significant uncertainty during the initial approach phase. The DemoSpeedUp method performs inaccurate labeling, such as misidentifying a high visual entropy in this section as a precision section or conversely preserving a section where acceleration is permissible. Consequently, the success rate drops to 1 / 20 in ACT-based systems. In contrast, the method of the present invention accurately identifies the point of approach stabilization based on the actual change in distance between the object and the gripper, and maintains a high success rate of 18 / 20 or 19 / 20 by correcting the semantic judgment of whether the section is a precision or non-precision section through semantic analysis. This means that the method of the present invention remains stable even in complex situations where the actual manipulated object is moving, providing much higher reliability compared to existing methods that rely on simple visual signals.

[0126] Overall, the method of the present invention provides the effect of increasing the overall work speed by significantly shortening the episode length while maintaining a much more stable success rate than DemoSpeedUp in all four real-world tasks. In both the ACT and DP policies, the average success rate improved to 90.0% and 85.4%, respectively, while the average episode length accelerated from 2.21× to 2.41×, demonstrating that the technical effect of “safely accelerating only non-precision sections while maintaining or improving the success rate,” which is the goal of the present invention, works effectively even in actual robot operation environments. In other words, Figure 9 clearly demonstrates that the method of the present invention provides results that significantly improve the practical usability of the imitation learning policy by stably maintaining precision operation sections that existing techniques could not accurately handle due to variability in the real-world environment, sensory noise, and initial approach instability.

[0127] FIG. 10 is a table of ablation experiment results showing the IoU of segment division and the number of detected segments by comparing the method of the present invention with the Insertion and Transfer Cube operations, the case where distance information (r) is removed, the case where scene description information (Description) is removed, and the case where both information is removed.

[0128] Specifically, FIG. 10 presents experimental results comparing the method of the present invention with three modified ablation conditions for Insertion and Transfer Cube operations, quantitatively showing the difference in the IoU, which indicates the accuracy of segment segmentation, and the number of detected keep segments. This table clearly shows, in particular, how the 3D distance information (r) and VLM-based scene description, which are key components of the present invention, each contribute to the correct detection of keep segments.

[0129] First, the IoU values ​​in the Insertion operation most clearly demonstrate the importance of these components; the IoU for the entire configuration of the invention (including both distance information and scene description) records the highest value at 0.5166, indicating that accurate segmentation is being achieved. Conversely, when distance information is removed, the IoU drops sharply to 0.0224, because the precise contact motion characteristic of the Insertion operation is mostly determined based on minute changes in distance between the gripper and the object. The precision stage in the Insertion operation is the frame-unit distance between two objects, as expressed in Equation 2. It is identified through the long-term decreasing trend and small-scale distance oscillations; when these key variables disappear, the model no longer has a basis to distinguish an imminent contact state. Therefore, the phenomenon where the IoU drops to near zero demonstrates that distance information r is essential for precise segment detection.

[0130] Even when scene descriptions are removed, the IoU drops significantly to 0.1024, suggesting that phase changes in an insertion operation are difficult to adequately explain by simple geometric distances alone, and that the LLM can interpret the meaning of the interval more reliably when semantic cues describing the operation context are provided. In particular, ESPADA is designed not to simply use arbitrary semantic descriptions, but for the LLM to interpret whether a point in time is "approaching," "aligning," or "departing after completion" based on scene descriptions generated by the VLM. When these semantic aids disappear, the interpretation of meaning becomes unstable even with changes in distance patterns, and as a result, the IoU also drops significantly.

[0131] A similar trend is observed in the Transfer Cube task. While the IoU of the entire model is 0.3064, removing scene descriptions lowers the IoU to 0.0584. This reflects the fact that, in this task, the various manipulation steps are spatially arranged in similar configurations, making it difficult to separate the intent of the operation or the meaning of the phases based solely on geometric features. Indeed, during the process of moving a cube, the distance between the gripper and the object often does not fluctuate significantly at any point during actions such as grabbing, lifting, moving, or placing down. Therefore, semantic descriptions help the LLM understand the purpose of each step and the context of the operation, playing a role in more consistently restoring boundaries that simple distance patterns cannot identify.

[0132] Meanwhile, regarding the number of keep chunks, the same number of chunks were detected in all four experiments, revealing that the precision and stability of the model cannot be judged solely by the “formal number” of chunks. Ultimately, what matters is how accurately the boundaries of each chunk correspond to actual precise movements and how well the semantic flow of the operation is preserved, even when the same number of chunks are generated. The fact that only the overall model consistently maintains a high IoU in this regard demonstrates that the components of the present invention work complementarily. That is, distance information provides the basis for quantitatively detecting phase changes, and semantic description enables high-level interpretation that distinguishes semantic differences even in complex scenes with the same distance pattern.

[0133] Consequently, the elimination experiment of FIG. 10 clearly demonstrates that while the components of the present invention are meaningful individually, the two elements combined can classify precise and non-precision segments most accurately. This is very important in that if incorrect segment division occurs, precise motions may be cut off during the subsequent downsampling process, or conversely, non-precision motions may be excessively preserved, leading to a weakening of the acceleration effect. Considering this, FIG. 10 can be considered quantitative evidence clearly demonstrating that the core design of the present invention is justified and that semantic and spatial awareness-based segment division is much more robust and generalizable than existing methods.

[0134] In one embodiment, the processor (120) may be configured to correct the reliability of the segment classification by combining at least one of the robot's joint state information or joint velocity information included in the demonstration data with the relationship information to improve the classification accuracy of the precision motion segment and the non-precision motion segment.

[0135] In this embodiment, the processor (120) takes into account that there are cases where the characteristics of the precision operation stage cannot be sufficiently reflected solely by the change in distance between the gripper and the object or the semantic analysis results extracted from demonstration data. Therefore, it enhances the stability of the precision section determination by additionally utilizing high-reliability proprioceptive information—such as signals like joint position, joint velocity, and joint acceleration—that can always be provided within the robot. In particular, the reliability of the classification is quantitatively corrected by utilizing the fact that the precision operation section exhibits characteristics where the distance signal is constant or changes slightly while the deviation in joint velocity is almost non-existent, whereas in the non-precision movement section, the change in joint velocity is relatively large even under the same distance conditions.

[0136] Joint state-based correction is performed by the processor (120) for each time point t of the demonstration data, joint velocity and change in joint angle It is performed by calculating and combining it with relationship information (r_t) to calculate the reliability. For example, the processor (120) may introduce a simple combined score function as follows.

[0137] [Mathematical Formula 6]

[0138]

[0139] Here is a segment importance function based on the rate of change of gripper-object distance by time point, and is a proprioceptive stability measurement function that reflects the magnitude and rate of change of joint movement. class is a weight for combining two information sources, and the processor (120) can dynamically adjust it by considering semantic analysis reliability, temporal trend of distance, characteristics of the robot model, etc. For example, in the section where the joint velocity is maintained below a certain threshold, Increase it to raise the probability of being classified as a precision section, and conversely, in sections with large joint velocity changes is reduced to determine the corresponding section as a non-precision section. Additionally, if distance information and joint information conflict, the results of semantic analysis are utilized as an additional decision-making factor. Maintain consistency in classification by recalibrating.

[0140] Such joint-based correction is particularly useful when the distance signal is temporarily inaccurate—for example, when an object is partially obscured from the camera's field of view or when the depth estimation model contains temporary noise. In this case, the processor (120) can prevent incorrect non-precision labeling by analyzing whether the joint velocity pattern exhibits typical characteristics of a "precision phase," such as small velocity changes, the absence of repetitive micro-vibrations, and the maintenance of a constant velocity. Conversely, in sections where the distance signal is static but the joint velocity is relatively large or the amount of change in direction is large, the processor (120) can recognize that the section is actually a fast approach or exit phase and determine to increase the likelihood of a non-precision section.

[0141] In another embodiment, the processor (120) may use a probabilistic judgment model to determine whether there is a precision interval as follows.

[0142] [Mathematical Formula 7]

[0143]

[0144] Here It enables probabilistic judgment using the sigmoid function, and is a weight that can be set through learning or empirical means. This model has a logical structure in which the probability of a precise segment increases as the distance signal gets closer and as the change in joint velocity and joint angle becomes smaller. The processor (120) combines this probability value with the existing semantic segment classification result to dynamically adjust the reliability of the final segment label. For example, if the semantic classification is judged to be precise but the joint-based probability model shows a high probability of being imprecise, the processor (120) marks the frame as a candidate segment and ultimately determines the label again by referring to the continuous pattern of surrounding frames.

[0145] Joint-based correction can also be extended to apply different weights depending on the robot's kinematic structure or the type of task. For example, in a robot arm composed of rotation centers, the amount of change in joint angle While it accurately reflects the detailed characteristics of this type of operation, in manipulators including a straight slider structure, joint speed The pattern of may be a more reliable signal. The processor (120) may automatically select a weight profile per task reflecting these characteristics, or pre-set weights based on the statistical characteristics of previous demonstrations.

[0146] Consequently, the correction process using joint state information or joint velocity information in this embodiment significantly improves the accuracy and stability of segment classification compared to a method that utilizes only distance-based relationship information and semantic analysis results, and can effectively compensate for boundary uncertainty arising from depth estimation noise or complex operation trajectories. This prevents problems where precise operation segments are unnecessarily accelerated or non-precision segments are excessively preserved, and consequently provides the effect of ensuring more stable and actual operation performance in subsequent downsampling and imitation learning policy learning.

[0147] FIG. 11 is a flowchart illustrating a method for improving the execution speed of a robot using spatial recognition-based demonstration data downsampling for imitation learning according to an embodiment of the present invention.

[0148] Referring to FIG. 11, a method for improving the execution speed of a robot according to one embodiment of the present invention may include the following steps S1 to S4. However, it is not limited thereto, and other general-purpose steps may be further included in the method for improving the execution speed of a robot.

[0149] The method for improving the execution speed of the robot described above may consist of steps processed sequentially in the robot execution speed improvement system (100) described above. Therefore, even if details are omitted below, the description of the robot execution speed improvement system (100) above may be equally applicable to the robot execution speed improvement method described below.

[0150] In the above S1 step, the processor (120) can obtain the two-dimensional positions of the object and the gripper from the video frames included in the robot's demonstration data, calculate three-dimensional coordinates corresponding to the two-dimensional positions using a monocular depth estimation model, and generate relationship information that expresses the change in spatial relationship between the object and the gripper by calculating the temporal rate of change and distance between the calculated three-dimensional coordinates.

[0151] In the above S2 step, the processor (120) inputs scene information acquired in frame units along with relationship information to perform semantic analysis, and based on the semantic analysis results, classifies each section of the demonstration data into a precise operation section or a non-precision operation section.

[0152] In the above S3 step, the processor (120) can apply a time axis alignment algorithm to one of the classified intervals to propagate the interval label of the reference interval to another interval of the demonstration data set.

[0153] In the above S4 step, the processor (120) can generate training demonstration data adjusted such that the amount of movement of the robot after downsampling is equal to the amount of movement of the robot in the demonstration data by downsampling the precise movement section according to the propagated label, maintaining the original density, and expanding the frame interval of the non-precision movement section.

[0154] Meanwhile, the method for improving the execution speed of a robot, which is performed by the robot execution speed improvement system (100), may be implemented in the form of a computer program stored on a computer-readable storage medium. That is, the computer program may include instructions for implementing the method for improving the execution speed of a robot, and the instructions of the computer program may be stored on a computer-readable storage medium. The computer program may include a mobile application.

[0155] For example, computer-readable storage media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute computer program instructions, such as ROM, RAM, and flash memory. Computer program instructions may include machine code generated by a compiler and high-level language code that can be executed by a computer using an interpreter, etc.

[0156] As described above, although the present invention has been explained by limited embodiments and drawings, the present invention is not limited thereto, and it is obvious that various modifications and variations are possible within the scope of the technical spirit of the present invention and the equivalent scope of the claims described below by those skilled in the art to which the present invention belongs.

[0157] Meanwhile, although terms indicating directions such as up, down, left, right, front, and back have been used in this invention, these terms are used merely for convenience of explanation, and it is obvious to those skilled in the art that they may vary depending on the position of the object or the position of the observer. Explanation of the symbols

[0158] 100 : System 110 : Memory 120 : Processor

Claims

Claim 1 Memory for storing robot demonstration data; The processor comprises a processor configured to execute instructions stored in the memory to obtain 2D positions of an object and a gripper from video frames included in the demonstration data, calculate 3D coordinates corresponding to the 2D positions using a monocular depth estimation model, generate relationship information expressing changes in the spatial relationship between the object and the gripper by calculating the temporal rate of change and distance between the calculated 3D coordinates, perform semantic analysis by inputting scene information obtained on a frame-by-frame basis along with the relationship information, classify each segment of the demonstration data into a precision motion segment or a non-precision motion segment based on the results of the semantic analysis, apply a time-axis alignment algorithm to one of the classified segments to propagate the segment label of the reference demonstration segment to other demonstration segments of the demonstration data set, and downsample the precision motion segments by maintaining the original density and the non-precision motion segments by expanding the frame interval according to the propagated label, thereby generating training demonstration data adjusted so that the amount of movement of the robot after downsampling is the same as the amount of movement of the robot in the demonstration data. A robot execution speed enhancement system configured to perform downsampling by configuring a set of iterative frames having multiple starting points for a non-precision motion section, and then selecting frames from each set at regular intervals. Claim 2 A robot execution speed enhancement system according to claim 1, wherein the processor is configured to calculate the 3D coordinates by back-projecting the 2D position and monocular depth value of the object and gripper into a 3D space corresponding to the intrinsic parameters of the camera. Claim 3 A robot execution speed enhancement system according to claim 1, wherein the processor is configured to analyze the distance between the calculated three-dimensional coordinates and the temporal change pattern of the distance, and classify each section of the demonstration data into a precision motion section when a distance reduction trend or minute fluctuation exists. Claim 4 A robot execution speed enhancement system according to claim 1, wherein the processor is configured to calculate the boundary between a precision operation section and a non-precision operation section by combining the semantic analysis result and the relationship information. Claim 5 A method for improving the execution speed of a robot performed by a robot execution speed improvement system comprises: a step of acquiring two-dimensional positions of an object and a gripper from video frames included in the robot's demonstration data, calculating three-dimensional coordinates corresponding to the two-dimensional positions using a monocular depth estimation model, and generating relationship information expressing changes in the spatial relationship between the object and the gripper by calculating the temporal rate of change and distance between the calculated three-dimensional coordinates; a step of performing semantic analysis by inputting scene information acquired on a frame-by-frame basis along with the relationship information, and classifying each segment of the demonstration data into a precision motion segment or a non-precision motion segment based on the results of the semantic analysis; and a step of applying a time-axis alignment algorithm to one of the classified segments as a reference demonstration segment to propagate the segment label of the reference demonstration segment to other demonstration segments of the demonstration data set. A method for improving the execution speed of a robot, comprising the step of generating training demonstration data adjusted such that the amount of movement of the robot after downsampling is equal to the amount of movement of the robot in the demonstration data, by downsampling the precision motion section according to the propagated label by maintaining the original density and expanding the frame interval of the non-precision motion section, wherein the robot execution speed improvement system is configured to perform downsampling by configuring a set of iterative frames having a plurality of starting points for the non-precision motion section and selecting frames at a certain interval from each set.

Images