Method and device for demonstration-based robot programming supplemented by video

Abstract

A method of programming an industrial robot, comprising: recording movements of a robot manipulator during a demonstration-based programming session, to obtain a robot trajectory (301); using a natural-language model (171), parsing speech data describing the programming session into at least one robot parameter (305) value and annotating the robot trajectory with the robot parameter value; and, on the basis of the annotated robot trajectory (306), generating a robot program (307). The method further comprises: capturing a video (302) of the programming session; using a vision-enabled language model, VLM (172), generating descriptions (303) of movements of the robot manipulator which are visible in the video; and generating said speech data (304) on the basis of the descriptions of the robot manipulator movements.

Description

METHOD AND DEVICE FOR DEMONSTRATION-BASED ROBOT PROGRAMMING SUPPLEMENTED BY VIDEOTECHNICAL FIELD

[0001] The present disclosure generally relates to the field of robotic control, and specifically to demonstration-based programming of an industrial robot. More precisely, methods and devices are proposed herein which an operator can utilize to generate a robot program from a recorded robot trajectory and a video of the programming session, wherein the program generation is guided by neural-network generated descriptions in natural language of the robot’s movements.BACKGROUND

[0002] Lead-through robot programming, once an area exclusive to seasoned programmers of high technical expertise, has seen a broadening user base lately thanks to the advent of Large Language Models (LLMs), and Generative Pre-trained Transformers (GPTs) in particular. LLMs are increasingly being applied to robot control and robot programming. See for instance the applicant’s prior disclosure PCT / EP2023 / 087949, which proposed a speech-supplemented robot programming method based on the following workflow:• A demonstration-based (in particular, a kinesthetic) programming session, during which movements of the robot manipulator are recorded and saved as a robot trajectory.• A recording of speech data, which takes place during the demonstrationbased programming session.• By suitable prompting, an LLM is caused to parse the speech data into values of robot parameters, such as a tool state, a movement speed, a degree of compliance with the trajectory, a degree of movement precision, a choice of reference frame to express the movements in.• The robot trajectory is annotated with each robot parameter value at a point of the trajectory which corresponds to the time of utterance.• A robot program is generated as a sequence of commands which realizes the robot trajectory while applying the robot parameter values as modifiers.The programming method according to PCT / EP2023 / 087949 offers the operator a convenient, hands-free way of adding explanatory remarks to the recorded movements (purpose of a step, aspects to pay particular attention to), or of requesting certain robot parameter settings which cannot be seen or sensed during the demonstration (speed, precision) etc., with the ultimate aim of providing a usable robot program in shorter time. The programming workflow as such appears to be proper to the applicant.

[0003] One potential concern is that a relatively quiet environment, where the speech data can be captured with few errors, maybe needed for this earlier proposal to reach optimal performance. Further, the method according to PCT / EP2023 / 087949 presupposes a reasonable multitasking ability from the operators. Newcomers may perceive the demonstration component as a considerable cognitive load in itself, which may to some extent stop them from providing a consistent spoken component. With practice, however, a motivated newcomer will be able to provide spoken explanations during a robot programming session that are consistent, comprehensive and in line with the person’s intentions.SUMMARY

[0004] One objective of the present disclosure is to overcome or at least mitigate the concerns related to the state of the art. A further objective is to propose useful methods and devices which an operator can utilize to generate a robot program from a recorded robot trajectory and a video of the programming session. A further objective is to shorten the learning time for a newcomer to start using a robot programming device productively. A further objective is to offload, from the operator, the relatively demanding task of providing a spoken explanation of a robot task while demonstrating this task. A further objective is to enable accurate demonstrationbased robot programming in a noisy environment.

[0005] At least some of these objectives are achieved by the invention as defined by the independent claims. The dependent claims relate to advantageous embodiments of the invention.

[0006] In a first aspect of the present disclosure, there is provided a method of programming an industrial robot, which comprises a robot manipulator and a robot controller. The method comprises: recording movements of a robot manipulator during a demonstration-based programming session, to obtain a robot trajectory; using a natural-language model, parsing speech data describing the programming session into at least one robot parameter value and annotating the robot trajectory with the robot parameter value; and on the basis of the annotated robot trajectory, generating a robot program. According to said first aspect, the method further comprises: capturing a video of the programming session; using a vision-enabled language model (VLM), generating descriptions of movements of the robot manipulator which are visible in the video; and generating said speech data on the basis of the descriptions of the robot manipulator movements.

[0007] According to the method of the first aspect, the robot parameter values are obtained by parsing speech data which - unlike in PCT / EP2023 / 087949 - is not recorded together with the robot manipulator movements. Instead, the speech data is derived from VLM-generated descriptions of the robot manipulator movements. The speech data may be regarded as a type of simulated (or predicted) vocal commands. The present inventors have established that these simulated vocal commands, despite being entirely based on knowledge incorporated in the VLM, have a sufficient information content to allow them to be parsed into meaningful robot parameter values for modifying robot commands. The inventors have obtained satisfactory performance with a generic VLM, not specifically trained for robot-related processing.

[0008] To the applicant’s knowledge, there are no reports of using a VLM for this purpose in the literature, much less in the programming workflow under consideration.

[0009] In some embodiments of the method of the first aspect, the keyframes are extracted from the video, and the descriptions of robot manipulator movements are restricted to intervals between such keyframes. This limits the computational burden on the VLM while maintaining acceptable performance. Further, the restriction to keyframes or keyframe intervals also brings with it an advantageous downsampling effect, which tends to eliminate vibrations, oscillatory, non-productive or non- transformative movements. In the context of demonstration-based robot programming, such movements on a short time scale are generally caused by randomor unintentional factors and thus have an inherently lower content of useful information. Several heuristics and useful rules for selecting keyframes for this purpose will be presented below.

[0010] In some embodiments, the method includes a step of eliminating adjacent descriptions if they are too similar. This limits the data volume to be input to the natural-language model and thus helps observe any token limit that may apply.[oon] In some embodiments, the method includes deriving speech data from the descriptions of robot manipulator movements in the form of commands for the robot manipulator to perform actions. The speech data is thus generally in the imperative mode than in the indicative mode. Experience shows that this form of speech data lends itself well to the parsing process, where robot parameter values are provided.

[0012] In some embodiments, the speech data is generated further on the basis of the recorded video. This is to say, the video is not only used as input to the VLM, from which the VLM generates descriptions of robot manipulator movements, but the video is also fed to the process of generating the speech data. This allows the contextual data from the appearance of the demonstration to be taken into account and / or possible ambiguities can be eliminated. The underlying structure defined by the movement descriptions is nevertheless respected.

[0013] In some embodiments, the method comprises an estimation of a current pose of a workpiece during the demonstration-based programming session.Demonstration-based programming may include kinesthetic programming and so- called passive observation. In kinesthetic robot programming, the robot handles the workpiece during the programming session. In passive observation (robot-less demonstration), a human operator who carries out the robot task may handle the workpiece in the robot’s stead, and the robot learns to imitate the handling which the workpiece undergoes. While in most realistic use cases it cannot be expected that the workpiece will carry position / orientation sensors, knowledge of the workpiece’s poses during the execution of the robot task is potentially very helpful in order to arrive at an accurate robot program. In particular, the composite pose-estimation algorithm from the applicant’s prior disclosure PCT / EP2024 / 061921 can be used to advantage.

[0014] In a second aspect of the present disclosure, there is provided a programming device for facilitating programming of an industrial robot, whichcomprises a robot manipulator and a robot controller. The programming device comprises memory and processing circuitry configured to perform the method of the first aspect.

[0015] The second aspect generally shares the effects and advantages of the first aspect, as reviewed above, and it can be put into practice with a corresponding degree of technical variation.

[0016] The present disclosure further relates to a computer program containing instructions for causing a computer, or the programming device in particular, to carry out the method according to the first aspect. The computer program may be stored or distributed on a data carrier. As used herein, a “data carrier” may be a transitory data carrier, such as modulated electromagnetic or optical waves, or a non-transitory data carrier. Non-transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of magnetic, optical or solid-state type. Still within the scope of “data carrier”, such memories may be fixedly mounted or portable.

[0017] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to “a / an / the element, apparatus, component, means, step, etc.” are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order described, unless this is explicitly stated.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, on which: figure 1 shows a work area, in which a robot manipulator operates under the control of a robot controller, and a programming device supporting demonstration-based robot programming; figure 2 is a flowchart of a robot programming method according to embodiments herein; andfigure 3 illustrates information flows during an execution of the method illustrated in figure 2.DETAILED DESCRIPTION

[0019] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, on which certain embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.System overview

[0020] As explained initially, the present disclosure relates to the field of industrial control and particularly to technologies for controlling industrial machines. Industrial machines include machines which are conventionally used in the example industry branches enumerated above, ranging from large / complex machines to small / simple machines, such as assembly machines, packaging / labelling machines, chemical machinery, papermaking machinery, agricultural equipment, energy generators and converters, and industrial robots. In the terminology of the present disclosure, “industrial robot” is used in a broad sense, to cover in particular manufacturing robots, material-handling robots, assembly robots, cutting / welding robots, service robots, collaborative robots, hygiene robots, industrial robot tracks, industrial robot positioners. An industrial robot may be a stationary robot or a mobile robot, such as an automated guided vehicle (AGV), an autonomous mobile robot (AMR) or an autonomous mobile manipulator robot (AMMR). The term industrial robot covers the full range from lightweight robots designed to replace human manual work, over collaborative robots for supporting a human worker, all the way up to heavy-duty robots.

[0021] By way of illustration and not limitation, figure 1 shows an industrial robot 100 made up of a robot manipulator no and a robot controller 120. The robot manipulator no and the robot controller 120 are joined by a wired or wireless bidirectional data connection, which conveys control signals, sensor data etc.

[0022] The robot manipulator no includes an arm, which extends from a base 112 and is made up of structural elements 114 and at least one linear or rotary joint 113. The arm may further include an end-effector 111 allowing it to carry tools by which it may interact with objects in the form of various workpieces 151, which are present in a work area 150 of the robot manipulator 110. The workpieces are subject to manufacture, processing or handling by the robot manipulator no. The work area 150 may further include non-workpiece objects 152, such as containers, fixtures, robot positioners, separators, insulators, supports etc. These objects 152 maybe generic or they may be specifically adapted to the workpieces 151 handled by the robot manipulator 110. Apart from wear, staining etc., they are generally in the same condition at the beginning and end of a work cycle. The arm of the robot manipulator no is movable by action of internal motors, drives or actuators (not shown), and it includes transducers, sensors and other measuring equipment (not shown), from which the manipulator’s no current position, pose, technical condition, load etc. can be derived, to some degree of accuracy. The position, pose etc. of the robot manipulator 110 may in particular refer to a point on the arm, particularly to a toolcenter point (TCP).

[0023] Figure 1 shows an implementation where the arm of the robot manipulator no is provided with an imaging device 117 (e.g., camera, video camera, depth camera, lidar, radar). More precisely, the imaging device is here mounted on the arm’s endeffector, which also carries the tool 111. To achieve this function, the imaging device 117 need not be mounted on the end-effector, but it may be sufficient for the imaging device 117 to be in rigid relationship with the end-effector. The imaging device 117 may be used for capturing a video of a demonstration-based programming session, as explained below, unless this task is allocated to a different imaging device 117.

[0024] The positions, poses etc. may be expressed with respect to one of multiple possible reference frames, including a first fixed reference frame O0defined with respect to a point in the work area 150, a second fixed reference frame (not shown) defined with respect to the base 112, a first reference frame 0 defined with respect to an initial position of a workpiece 151, or a second reference frame 02defined with respect to a position of a non-workpiece object 152 in the work area 150. Here, the first fixed reference frame O0is defined with respect to the point in the work area 150 in the sense that its origin is situated in that point. Alternatively, a reference framemay be defined with respect to a reference point in the sense that its origin is situated a predefined translation from the reference point. The first and second fixed reference frames, as well as any further reference frames which are independent of the positions of objects 151, 152 in the work area 150, maybe collectively referred to as neutral reference frames. All reference frames shown in figure 1 have a common orientation, i.e., their respective first, second and third axes are parallel. Without departing from the scope of the present disclosure, a further option is to use reference frames with mutually different orientations, including a reference frame which is oriented in accordance with a pose of an object 151, 152 in the work area 150 or another suitable reference objects. In particular, one may use a reference frame which has its origin situated in the TCP and is oriented parallel to the tool 111 of the manipulator no at all times.

[0025] The robot controller 120 comprises processing circuitry 121, a memory 122 and a communication interface 123. Example content of the memory 122 during operation may includeS: an operating system, basic settings, software implementing generic movements, sensing, self-monitoring, generically useful functionalities and services (all typically contributed by an original manufacturer), task- or role-specific configurations, configuration templates (typically contributed by a robot system integrator), and site-specific settings (typically contributed by an end user);C: robot programs or projects causing the robot manipulator no to perform useful or intended tasks in its work area 150.Processes that execute the robot programs may do so in accordance with the program-independent memory content S, e.g., by making calls to available functionalities, libraries, routines or parameter values therein. It is recalled that the memory 122 and processing circuitry 121 of the robot controller 120 maybe distributed and / or contain networked resources having a different physical localization than figure 1 suggests.

[0026] Each of the programs C may contain a plurality of movement instructions relating to locations such as points, poses, paths as well as modulated paths. A program maybe a compiled executable (binary) or a script. A movement instruction relating to a modulated path may be expressed as - or may include - a process-on-path instruction. The programs C maybe created by an operator 190 with the aid of a robot programming device (programming station) or a general-purpose computer, or they may be created directly at the robot controller 120 if it has an operator interface. In the first two cases, versions of the programs C may be downloaded to the robot controller 120 over a wired or wireless connection or by being temporarily stored on a portable memory. A robot program maybe created by means of demonstration-based programming according to the teachings in the applicant’s prior disclosures PCT / EP2023 / 087949 and PCT / EP2024 / 066288.

[0027] As used herein, “demonstration-based” robot programming (or programming by demonstration [PbD], or programming from demonstration) includes a step where the operator performs a portion of a robot task or guides the robot manipulator such that it performs a portion of this robot task. The operator’s activities are imaged or, in other ways, observed by a programming device. Demonstration-based robot programming includes kinesthetic programming as a special case. The term kinesthetic programming - by allusion to the robot’s proprioceptive ability to sense its own position, pose and movements - is a programming approach in which the operator physically moves the robot manipulator to imitate the desired motions. The terms kinesthetic programming and lead-through programming are synonymous or at least partially overlapping in meaning. The state of the robot during a kinesthetic programming session is typically recorded by means of the robot’s onboard sensors, e.g., joint angles and torques. As used herein, kinesthetic programming also includes teleoperation as a special case, where the movement of the robot manipulator is controlled by an external input to the robot through a joystick, graphical user interface or other input means; kinesthetic programming by teleoperation does not require the operator to be present in the work area of the robot.

[0028] A dedicated programming device 160 is shown in the right-hand portion of figure 1. From the programming device 160, the created programs C can be transmitted to the communication interface 123 of the robot controller 120 and then stored in the memory 122 where they are available for execution. It is noted that the programming device 160 may be implemented as a set of collaborating components within the robot controller 120. In fact, the components can be shared with the robot controller 120, e.g., by using the processing circuitry 121 for the dual purposes of robot control and programming and / or using the memory 122 for the same dualpurposes. In other words, the programming device 160 may constitute a portion of a multi-purpose device; it need not be a standalone device or a device with programming as its sole or main purpose.

[0029] The programming device 160 - which is shown as a standalone device in the non-limiting example of figure 1 - comprises processing circuitry 161, memory 162 with executable software 163, and at least one communication interface 164, 165. The programming device 160 may further comprise at least one operator interface 166, in particular a graphical operator interface or graphical user interface (GUI).

[0030] The programming device 160 further has access to an imaging device (e.g., camera, video camera, depth camera, lidar, radar) 167, by which images or video of the demonstration-based programming session can be captured. In particular, the imaging device may be a depth camera, lidar or radar, which in addition to - or instead of - the two-dimensional appearance of an object determines a depth coordinate of the object, e.g., by time-of-flight measurements, triangulation, reflection or other per se known techniques. An RGB-D camera is an example of a depth camera. The imaging device 167 maybe a part of the programming device 160, or it maybe integrated into the industrial robot 100 and optionally be used for other tasks as well. More generally, an equivalent function may be achieved by any imaging device which is in rigid relationship with the robot base 112.

[0031] The programming device 160 is optionally enabled to capture speech data, which could represent utterances or narration by the operator 190 during the programming session. The programming device 160 may for this purpose utilize one or more acoustic transducers (microphones) 168 arranged in the vicinity of the operator’s 190 normal position. Alternatively, the speech data maybe captured by a portable microphone or a headset worn by the operator 190. To parse speech data, the programming device 160 may utilize a natural-language model 171, which may be either stored internally or - as in the example configuration shown in figure 1 - may be available from a host computer or an external memory 170.

[0032] The natural-language model 171 stored in the external memory 170 may constitute a large language model (LLM), that is, a type of machine-learning algorithm which has been trained on very large datasets using deep learning techniques to be able to perform natural-language processing (NLP) tasks. Example NLP tasks are recognizing, summarizing, translating, predicting and generatingplausible textual content. A very large dataset in this sense may include of the order of one million parameters, such as tens of millions of parameters, such as hundreds of millions of parameters. An LLM may have a transducer architecture, particularly a transducer architecture with four cascaded key links when it processes the input data, namely: word embedding, position encoding, self-attention mechanism, feedforward neural network. At the time of filing this disclosure, noteworthy example LLMs include Bidirectional Encoder Representations from Transformers (BERT), Bard, BLOOM, Claude 2, various versions of Generative Pre-trained Transformer (GPT), Llama, PaLM 2, RoBERTa, T5, LaMDA, Turing NLG, Gemini 1.0. LLMs include, as a special case, multimodal models, such as Gemini 1.5. One benefit of an LLM is that the vocabulary is practically open-ended. The operator 190 can start using it without prior training. The demonstration-based programming can be carried out without requiring the operator to be extremely focused on using the right command words (or avoiding them) and / or syntax.

[0033] The external memory 170 further stores a vision-enabled language model (VLM) 172. In alternative terminology, the VLM 172 maybe referred to as a visionlanguage model. A VLM is machine-learning algorithm which has been trained on very large datasets to be able to perform processing which inputs or outputs combinations of textual and visual information. (It is emphasized that the training datasets need not be related to robot operation or to robot tasks, but the inventors have obtained advantageous results using generic VLMs.) A VLM is a multimodal model. A VLM may in particular be a vision-enabled LLM which, in addition to the above-reviewed characteristics of an LLM, is suitable for vision-centric tasks. Example types of VLM are contrastive-based models, models with masking objectives, generative-based models, or models from pretrained backbones. At the time of filing this disclosure, noteworthy example VLMs include versions of DALL-E, GPT-4 Vision, GPT-4 Turbo, GPT-40 and Gemini 1.5. At least DALL-E has functionalities for generating images from textual information (this is not a mandatory for implementing the present invention), and the further example models have functionalities for analyzing and understanding input images.

[0034] During a demonstration-based programming session, the programming device 160 has access to position data representing an actual position, a recorded position or recorded movements of the robot manipulator no, as well as images orvideo of the robot programming session. The position data may for example be obtained through the intermediary of the robot controller 120, which monitors position data in the normal course of its operation; alternatively, the programming device 160 is granted access to corresponding signals from the transducers, sensors or other measuring equipment in the robot manipulator no.Robot programming method

[0035] Turning to the flowchart in figure 2, embodiments of a method 200 of programming an industrial robot 100 of the type depicted in figure 1 will now be described. Not all steps shown in figure 2 are necessarily carried out in all embodiments. The output of the method 200 includes a robot program C to be executed by the robot controller 120 of the industrial robot 100 in figure 1. The robot program C maybe executable only by the robot controller 120 of the industrial robot 100 for which the programming method 200 was carried out. In some embodiments of the method 200, the robot program C can be executed also by further robot controllers of the same model as - or compatible with - the one for which the programming method 200 was carried out.

[0036] A robot program C may include a sequence of robot commands that cause the robot manipulator no to reproduce the trajectory. The robot commands may be selected from a predefined set of robot commands executable by the robot controller 120, such as commands compliant with the RAPID™ robot programming language.

[0037] The method 200 in figure 2 may be executed on a general-purpose processor and on the programming device 160 in particular. As mentioned, the programming device 160 may constitute a portion of a multi-purpose device. The method 200 may be considered to be a description of the behavior which the programming device 160 is configured for when active. The method 200 may as well correspond to the behavior of the robot controller 120 in a special programming mode, which the robot controller 120 can be requested to enter. Instructions for causing a computer - or the programming device 160 in particular - to carry out the method 200 maybe provided in the form of a computer program 164 (see figure 1).

[0038] The functioning and other characteristics of the method 200 may be better understood from the illustration in figure 3 of the information flows which occurduring an execution of the method 200. Here, in addition to the reference numbers from figures 1 and 2, the following notation is used:301 robot trajectory302 video303 descriptions of robot manipulator movements304 speech data305 robot parameter value306 robot trajectory307 robot program

[0039] A first step 210 of the method 200, which is executed during a demonstration-based robot programming session, includes recording movements of the robot manipulator no. On the basis of the recorded movements, a robot trajectory can be obtained, e.g., by combining the recorded movements. The robot trajectory 301 thus obtained may indicate the robot manipulator’s 110 position and / or pose (e.g., in Cartesian or joint-space coordinates) as a function of time. The position or pose need not be indicated for all points in time. Alternatively, the trajectory maybe expressed as a number of reference times at which the robot manipulator no is to assume corresponding setpoint positions or setpoint poses, wherein the robot manipulator 110 is free to have arbitrary positions or poses in the intervals between the reference times.

[0040] In a second step 211, a video (or video recording, or video sequence) 302 of the programming session is captured. The video 302 maybe captured by means of the fixed imaging device 167 or by an imaging device 117 in rigid relationship with the end-effector 111 of the robot manipulator no. Each frame of the video 302 maybe a visual image, a depth image or a combination of these. For the purposes of the present method 200, it is not essential that the video 302 have a high frame rate (e.g., a frame rate which is capable of capturing naturally-looking body movements). Rather, the video 302 maybe a sequence of sparse, even irregularly spaced, images of the work area 150.

[0041] In an optional step 212, a plurality of keyframes is extracted from the video 302. The keyframes maybe a subset of all video frames of a video sequence. Theextracted keyframes may serve as a way of restricting the generating of robot manipulator movement descriptions, as will be described next under step 213. Each keyframe thus extracted may be associated with a timestamp from which the keyframe’s position in the video can be derived. The timestamp may indicate a time of capturing of the keyframe, or a sequence of the frame within the video 302.

[0042] The inventors foresee various rules for identifying the keyframes.

[0043] According to one rule 212.1, a video frame shall be identified 212 as a keyframe if it depicts a significant visual change and / or it follows immediately after a significant visual change. A visual change may correspond to a local minimum or local decrease in a similarity of successive frames. Here and elsewhere, successive frames may be consecutive frames (with sequence numbers n, n + 1, n + 2, ...) or frames which are consecutive after subsampling down to every A th frame (with sequence numbers n, n + N, n + 2N, ...), where N > 2. In the second case, accordingly, the keyframes are a sub-selection from the 1: A-downsampled video. Either way, a visual change may be determined on the basis of a similarity metric.

[0044] According to another rule 212.2, the similarity metric is a structural similarity index measure (SSIM). More precisely, a video frame shall be identified 212 as a keyframe on the basis of a structural similarity index measure (SSIM) values for pairs of successive frames. For example, one may study the evolution of a function / (n) = l - 55 / M(Xn,Xn-1), where SSIM Xn, Xn_f) is the SSIM of frame n relative to frame n — 1, where Xn, Xn-denote the respective image data. The peaks of function (n) maybe interpreted as significant visual changes, hence suitable to extract as keyframes. SSIM maybe defined as a function of the respective pixel sample means, the respective pixel variance values and the covariance of the two images. See for example Wang, Z. et al., “Multiscale structural similarity for image quality assessment”, The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol. 2, pp. 1398-1402.

[0045] According to another rule 212.3, a video frame shall be identified 212 as a video frame as a keyframe if it depicts a perspective change and / or it follows immediately after a perspective change. This rule maybe particularly useful when the video 302 is captured using an image sensor 117 moving with the end-effector, as a perspective change will generally correspond to a movement of the robot manipulatorno. A perspective change maybe detected by means of a similarity metric, such as the SSIM. A perspective change may alternatively be detected by processing the video302 by a suitable positioning algorithm relative to a visual reference point in the work area 150, wherein an apparent movement of the visual reference point correspond to a perspective change.

[0046] In a next step 213 of the method, a vision-enabled language model VLM 172 is given the video 302 as input and is prompted to generate descriptions 303 of such movements of the robot manipulator no which are visible in the video 302. The descriptions 303 maybe expressed in natural language. Alternatively, the descriptions 303 may be expressed in latent space of the VLM 172, e.g., in the form of a bitstring or number which can be transferred to the next processing step 214. In embodiments where the natural-language model 171 and VLM 172 are implemented as a single multi-modal model, the latent space is common to the natural-language model 171 and VLM 172, and the latent-space representation can be made available internally, without a need for conversion into a transferable format. A benefit of expression the movement descriptions 303 in latent space is a possible reduction of the influence from random variations in wording and vocabulary, which are inherent in natural language, so that ultimately the annotations of the trajectory become more deterministic.

[0047] Optionally, the movement descriptions 303 may have timestamps, such that they can be associated with a corresponding point in the trajectory 301. Example VLMs were reviewed above.

[0048] In some embodiments, step 213 is restricted to a set of extracted keyframes (cf. step 212), in the sense that movement descriptions 303 are generated for the keyframes but not the remaining frames. In particular, the movement descriptions303 maybe generated for the intervals between the extracted keyframes, wherein the VLM 172 is prompted to describe the movement between a current keyframe and the preceding keyframe. Restricting step 213 to keyframes (or keyframe intervals) limits the computational burden on the VLM 172 while enabling acceptable performance to be upheld. Further, the restriction to keyframe (or keyframe intervals) also acts as a downsampling or low-pass filtering. This tends to eliminate movements on a short time scale, which generally have a lower content of useful information. If thekeyframes have been extracted 212 in such manner that they include timestamps, step 213 may include transferring the timestamps to the movement descriptions 303.

[0049] Step 213 optionally comprises a substep 213.1 of removing any movement description 303 if it too similar to an immediately preceding description 303. In particular, the movement description for frame n may be removed if its similarity with the movement description for frame n — 1 exceeds a threshold. This removal or duplicate or quasi-duplicate descriptions reduces the data volume to be input to the natural-language model and thus helps observe any token limit that may apply.

[0050] In a next step 214, based on the descriptions 303 of the robot manipulator movements, speech data 304 is generated. One option is to use the descriptions 303 as the speech data 304, i.e., to perform no processing or processing affecting the content of the descriptions 303. Step 214 may optionally include synthesizing the descriptions into a representation of speech, e.g., text annotated with prosodic or other features of natural spoken language, which are sampled from a statistical distribution. The speech data 304 need not be an audio track.

[0051] To facilitate temporal alignment with the recorded robot trajectory 301, the speech data 304 is preferably provided with time indications (time stamps) which can be associated with corresponding time indications (e.g., reference times) in the robot trajectory 301. Alternatively, the speech data 304 is decomposed into a plurality of phases, which are consecutive in time. As described in PCT / EP2023 / 087949, an LLM maybe used to decompose the speech data 304 into phases which correspond to respective events in the programming session or respective stages of a robot task.

[0052] In some embodiments of the method 200, the speech data 304 have the form of natural-language commands for the robot manipulator no to perform actions. For example, if a motion description indicates that some workpieces 151 have been arranged in regular rows, the corresponding natural-language command could be to line up all workpieces 151 with a suitable spacing.

[0053] In some embodiments of the method 200, the speech data 304 is generated 214 further (directly) on the basis of the video 302. In particular, an interpolation method may be used that evenly distributes the movement descriptions 303 between keyframes with timestamps, as will be discussed below in the Examples section.

[0054] An optional next step 215 of the method 200 contains an estimation of one or more poses which a workpiece 151 had during the programming session. The estimation is based on what is visible in the video 302. The estimated poses may be fed into the generation of the robot program 307 to increase the accuracy or eliminate ambiguities.

[0055] To implement the optional step 215, when the video 302 is a depth video, one may in particular apply a composite pose-estimation algorithm, as described in detail in the applicant’s earlier disclosure PCT / EP2024 / 061921. The composite pose estimation algorithm may be executed locally or on a processing resource outside the programming device 160, such as a networked (‘cloud’) processing resource. The pose estimation algorithm maybe implemented as a combination of a video-object segmentation (VOS) sub-algorithm configured to determine a mask of a visual object in an image, and an object-pose tracking (OPT) sub-algorithm configured to track a pose of a visual object over multiple depth-video frames.

[0056] In a further step 216 of the method, a natural-language model 171 is prompted to parse the speech data 304 describing the programming session into at least one robot parameter value 305. The robot trajectory is then annotated with the robot parameter value 305. The robot parameter maybe, for example, a state of a tool carried by the robot manipulator, a movement execution parameter, a degree of compliance with the robot trajectory, a degree of movement precision, a reference frame, a drive system parameter. The annotation shall be local in the sense that it does not apply - at least not by default - to the full robot trajectory.

[0057] The output of step 216 may include the extracted robot parameter values associated with the respective phase of the speech data 304. The output is to be provided in a form that can be handed over safely to the downstream processing steps without losing or corrupting the information. The output is preferably organized phase by phase, that is, the set of robot-parameter assignments for each new phase are contained in a new item in the output. In particular, the extracted robot parameter values may be formatted in accordance with a data serialization format segmented into phases. Examples data serialization formats are JSON (specified in The JSON Data Interchange Syntax, Standard ECMA-404, 2ndedition (2017-12)), YAML (specified in YAML Ain’t Markup Language (YAML™), version 1.2, revision1.2.2 (2021-10-01)) and XML (specified in XML Signature Syntax and Processing, version 1.1, W3C Recommendation (2013-04-11)).

[0058] The natural-language model 171 used for carrying out step 216 may be an LLM. It is understood that the natural-language model 171 and the VLM 172 maybe implemented as distinct models, as shown in figure 1. Alternatively, the naturallanguage model 171 and the VLM 172 are implemented as a single multimodal model.

[0059] Then, in step 217, a robot program 307 is generated on the basis of the annotated robot trajectory 306 in accordance with the robot parameter values 305 with which the trajectory 301 has been annotated. The robot program 307 may be expressed as a sequence of robot commands (e.g., RAPID™ commands) which, when executed by the robot controller 120, cause the robot manipulator no to realize the trajectory 301 while applying the robot parameter values as modifiers to these robot commands.

[0060] Step 217 may include a first substep of sampling the trajectory into a sequence of discrete points and a second substep of selecting robot commands that cause the robot manipulator 110 to move between each pair of consecutive discrete points. The sampling used in the first substep may be time-uniform sampling (constant step duration), space-uniform sampling (constant step length), or a non- uniform sampling algorithm with controlled deviation, such as Ramer-Douglas- Peucker. In a RAPID™ environment, the second substep may be performed so as to output instances of the command MoveL (cartesian linear motion) or MoveJ (jointspace linear motion) or a combination of these. Optionally, the step 217 may include a postprocessing substep applied to the sequence of generated robot commands, such as formatting the sequence into a predefined script format by appending a header, performing a consistency check, or the like.

[0061] Optionally, step 217 may include taking the estimated poses of the workpiece 151 into account when generating the robot program 307. The poses may be used as a consistency check or for eliminating ambiguities. The poses may as well be used to refine the accuracy of gripping commands or processing commands. In some implementations, the estimated poses may appear explicitly in the annotations 305, e.g., to allow a verification that the robot program is executing in the way expected, so that irregularities can be discovered early. Alternatively, the estimatedworkpiece poses may have an indirect influence, such as preferable ways of gripping or lifting the workpiece 151 using the robot manipulator’s no end-effector 111.Example

[0062] In an example, an ABB GoFa™ robot was taught to lift a Lego™ block and place it into a bowl. The teaching was carried out by demonstration-based robot programming, wherein operator-guided movements of the robot manipulator were recorded (realization of step 210) but the operator did not provide any spoken input. Further, a depth video of the demonstration-based robot programming section was recorded (step 211) using a depth camera, which had been mounted to the endeffector of a robot manipulator. To ensure that the mounting has a good fit, a block adapted to the shapes of the depth camera and the end-effector was produced by 3D- printing and interposed between these.

[0063] In the example, keyframes extracted from the video 302 (step 212) were encoded in base64 format. The keyframes were extracted based on motion analysis and were thus irregularly spaced. They carried the following timestamps: 3.0 s, 11.0 s, 18.0 s, 27.0 s, 30.0 s, 32.0 s, 36.0 s, 41.0 s, 45.0 s, 52.0 s, 55.0 s, 57.0 s.

[0064] To execute step 213, a VLM 172 was given the example prompt in Table 1, where a portion of the base64 representation of the keyframes has been omitted.

[0065] An example response from the VLM 172, detailing each keyframe and its description, is found in Table 2.

[0066] Based on timestamps of the keyframes fed to the VLM 172, the movement descriptions are dated and collected in a summary output file, as shown in Table 3.

[0067] In a realization of step 214, speech data 304 in the form a timestamped summary of the video 302 was generated. The summary includes simulated vocal commands, which replace those vocal commands which were uttered by the operator in the applicant’s earlier disclosure PCT / EP2023 / 087949. The following interpolation algorithm was used:- Assign an end time for each sentence. For each keyframe description in the robot motion summary received from the VLM 172, assign the corresponding keyframe time as the end time for that sentence. The keyframe time can be taken from the video 302.- Assign a start time for each sentence. If the corresponding keyframe is the first one, the start time is set to 0:00. Otherwise, assign the start time slightly after the previous keyframe time.- Assign a start time for each word. Linearly interpolate the words of a sentence from the start time to the end time of the sentence.- Assign an end time for each word. Considering the average speaking rate is approximately 150 words per minute in English, each word takes about 0.4 seconds to utter. The default duration of each word is therefore set to0.4 seconds. Adjust the end time for each word if there is time overlapping, i.e., if 0.4 seconds is too long for a word, set the end time slightly before the start time of the next word.- Save the speech data 304 in JSON format. Table 4 shows an example of speech data in this format, wherein the portions relating to the second and further sentences have been omitted.This algorithm maybe said to generate the speech data 304 while applying the video 302 directly, i.e., not only by the intermediary of the movement descriptions 303.Using the simulated vocal commands, the prompt message in Table 5 was generated and sent to the LLM 171 (realization of step 216). The LLM 171 was thereby prompted to segment the timestamped text into phases, each phase annotated with a start and end time, to identify values of specified robot parameters, including speed, precision, exact_J allowing and reference^frame, and finally to format the output in accordance with the YAML format.A response from the LLM was received, outlining the sub-steps of a task demonstration and providing appropriate robot parameters, as shown in Table 6.Based on this step description, RAPID™ code for the demonstrated task was automatically generated. Excerpts of the resulting pick-and-place robot program is shown in Table 7, wherein the somewhat unwieldy initial definitions of constant waypoints (robtargets) Pick_and_Plac wobj_aug_legoo, Pick_and_Plac wobj_aug_bowlo, robtarget Pick_and_Plac pi etc. have been omitted for space reasons.The RAPID code was then tested on the GoFa™ robot to satisfaction. During the test, the positions of the block and bowl were allowed to differ from their positions in the demonstration phase. The RAPID code adapted to the new setup automatically and successfully performed the task. It is noted that none of the tested properties were intrinsically linked to the pick-and-place task, so that it maybe expected that robot programs for other robot tasks can be generated by the same approach (e.g., continuous manipulation, such as opening a box or drawer), possibly after minor adaptations within the capabilities of those skilled in the art.

[0068] The aspects of the present disclosure have mainly been described above with reference to a few embodiments and an example. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

Claims

CLAIMS1. A method (200) of programming an industrial robot (100), which comprises a robot manipulator (no) and a robot controller (120), comprising: recording (210) movements of the robot manipulator during a demonstration-based programming session, to obtain a robot trajectory (301); using a natural-language model (171), parsing (216) speech data (304) describing the programming session into at least one robot parameter (305) value and annotating the robot trajectory with the robot parameter value; on the basis of the annotated robot trajectory (306), generating (217) a robot program (307), characterized by further capturing (211) a video (302) of the programming session; using a vision-enabled language model, VLM (172), generating (213) descriptions (303) of movements of the robot manipulator which are visible in the video; and generating (214) said speech data on the basis of the descriptions of the robot manipulator movements.

2. The method (200) of claim 1, wherein the video (302) of the programming session is captured by means of a fixed image sensor (167) or an image sensor (117) in rigid relationship with an end-effector (111) of the robot manipulator.

3. The method (200) of any of the preceding claims, further comprising extracting (212) keyframes from the video, wherein the generating (213) of descriptions (303) of robot manipulator movements is restricted to intervals between the extracted keyframes.

4. The method (200) of claim 3, wherein said extracting (212) comprises identifying (212.1) a video frame as a keyframe if it depicts a significant visual change and / or it follows immediately after a significant visual change.

5. The method (200) of claim 3 or 4, wherein extracting (212) comprises identifying (212.2) a video frame as a keyframe on the basis of a structural similarity index measure, SSIM, values for pairs of successive frames.

396. The method (200) of any of claims 3 to 5, wherein said extracting (212) comprises identifying (212.3) a video frame as a keyframe if it depicts a perspective change and / or it follows immediately after a perspective change.

7. The method (200) of any of claims 3 to 6, wherein said extracting (212) comprises associating each keyframe with a timestamp.

8. The method (200) of any of the preceding claims, wherein generating (213) the descriptions (303) of the robot manipulator movements comprises eliminating (213.1) a description if its similarity with an immediately preceding description exceeds a threshold.

9. The method (200) of any of the preceding claims, wherein the speech data (304) is generated (214) in the form of commands for the robot manipulator (120) to perform actions.

10. The method (200) of any of the preceding claims, wherein the speech data (304) is generated (214) further on the basis of the video.

11. The method (200) of any of the preceding claims, further comprising estimating (215), based on the video, one or more poses of a workpiece (151) during the programming session, wherein the robot program (307) is further generated (217) based on said estimated poses.

12. The method (200) of claim 11, wherein the video (302) of the programming session is a depth video, and the workpiece poses are estimated (215) by a composite pose-estimation algorithm, including a video-object segmentation, VOS, sub-algorithm configured to determine a mask of a visual object in an image, and an object-pose tracking, OPT, sub-algorithm configured to track a pose of a visual object over multiple depth- video frames, wherein the pose-estimation algorithm is configured to input a depth video, from which frames are extracted and fed to the VOS sub-algorithm, which determines respective object masks to be used by the OPT sub-algorithm alongside the depth video.4013- The method (200) of any of the preceding claims, wherein: the natural-language model (171) and the VLM (172) are implemented as distinct models; or the natural-language model and the VLM are implemented as a single multi-modal model.

14. A programming device (160) for facilitating programming of an industrial robot (100), which comprises a robot manipulator (no) and a robot controller (120), the programming device comprising memory (162) and processing circuitry (161) configured to: record movements of the robot manipulator during a demonstration-based programming session, to obtain a robot trajectory (301); using a natural-language model (171), parse speech data (304) describing the programming session into at least one robot parameter (305) value and annotating the robot trajectory with the robot parameter value; on the basis of the annotated robot trajectory (306), generate a robot program (307), characterized in that the processing circuitry is further configured to: capture a video (302) of the programming session; using a vision-enabled language model, VLM (172), generate descriptions (303) of movements of the robot manipulator which are visible in the video; and generate said speech data on the basis of the descriptions of the robot manipulator movements.

15. A computer program (163) comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method (200) of any of claims 1 to 13.

Images