Deployment of robotic system utilising augmented reality

Abstract

A robotic system and method of deploying a robotic system, where a sensor system obtains sensor data indicating properties of the robotic system, an AR (Augmented Reality) device comprising a display device displays a virtual environment established based on the sensor data, where a user associate context to parts of the virtual environment via the AR device. The context associated to parts of the virtual environment is then used to deploy the robotic system.

Description

DEPLOYMENT OF ROBOTIC SYSTEM UTILISINGAUGMENTED REALITYTECHNICAL FIELD

[0001] This specification describes examples of systems and processes for programming and setting up robotics systems to perform automated tasks utilizing augmented reality.BACKGROUND

[0002] A robot, such as a robotic arm, is configured to control an end effector to interact with the environment. An example end effector is an accessory or tool that the robot uses to perform an operation.

[0003] An example robotic arm is a computer-controlled robot that is capable of moving in multiple degrees of freedom. The robotic arm may be supported by a base and may include one or more links interconnected by joints. The joints may be configured to support rotational motion and / or translational displacement relative to the base. A tool flange may be on the opposite end of the robotic arm from the base. The tool flange contains an end effector interface. The end effector interface enables an end effector to connect to the robotic arm. In an example operation, the joints are controlled to position the robotic arm to enable the end effector to implement a predefined operation. For instance, if the end effector is a welding tool, the joints may be controlled to position, and thereafter to reposition, the robotic arm so that the welding tool is at successive locations where welding is to be performed on a workpiece.

[0004] Robots are typically installed in a robotic system, also referred to as a work cell comprising a set of machines that are organized to carry out one or more specified tasks. A robotic system can for instance be configured to carry out tasks such as, CNC machine tending, assembly, handling, palletizing, welding, finishing, inspection etc. Improving robot deployment in a work cell is an important aspect towards speeding up robot-based automation in manufacturing, as deployment of the work cell is both time consuming and difficult.SUMMARY

[0005] The objective of the present invention is to address the abovedescribed limitations with the prior art or other problems of the prior art. This is achieved by a robot system and method for deploying the robot system according to the independent claims.

[0006] The robotic system comprises: a sensor system configured to obtain sensor data indicative of one or more properties of the real environment of the robotic system;• an AR. (Augmented Reality) device comprising a display device; the AR device is configured to:• establishing the relative position and orientation of the AR device in relation to a spatial reference system of the robotic system;• establishing a virtual environment based on sensor data obtained by the sensor system;• displaying the virtual environment on the display device based on the relative position and orientation of the AR device in relation to the spatial reference system of the robotic system;• receiving inputs from a user enabling the user to select parts of the virtual environment and associating context to the selected parts;• a control system comprising one or more processing devices, the one or more processing devices being configured to control the robotic system based on the selected parts of the virtual environment and the context associated with the selected parts of the virtual environment.

[0007] The method of deploying a robotic system, comprising the steps of:• obtaining sensor data indicative of one or more properties of the real environment of the robotic system using at least one sensor system;• establishing a virtual environment based on sensor data obtained by the sensing device;• providing at least one AR (Augmented Reality) device comprising a display device;• establishing the relative position and orientation of the AR. device in relation to a spatial reference system of the robotic system;• displaying the virtual environment on the display device based on the relative position and orientation of the AR device in relation to the spatial reference system of the robotic system ;• selecting parts of the virtual environment via the AR device;• associating context to the selected parts of the virtual environment;• deploying at least a part of the robotic system based on the selected parts of the virtual environment and the context associated with the selected parts.

[0008] The robot system and method of deploying a robot system makes it possible to easily and faster to deploy the robot system as the Augmented Reality device makes assist the persons deploying the robotic system to provide context of parts of the robotic system which are used to deploy the robotic system. Aspects, advantages, and benefits of the present invention are described in the detailed description of the invention.

[0009] The present invention utilizes robotic knowledge of the system and context awareness of the user and intersect them with an Augmented Reality interface. The user simply takes a 3D scan of the robotic system with an AR device, for instance in form of a smart phone, adds context to parts of the robotic system that otherwise would be difficult to infer from the system and deploy the robotic system based on the 3D scan of the robotic system and the added context. The robotic system and method according to the present invention improves deployment time compared to the prior art methods using offline simulations tools.

[0010] The dependent claims describe possible embodiments of the method according to the present invention.DESCRIPTION OF THE DRAWINGSFig. 1 is a schematic illustration of a robotic system according to the present invention; fig. 2 illustrates a flow diagram of a method of deploying a robotic system according invention;fig. 3 illustrates a gray scale image of the experimental robotic system used for illustrating and testing the robotic system and the method according to the present invention figs. 4a, 4b, 5a, 5b, 6a, 6b, 7a, 7b, 8 illustrate exemplary views of a virtual environment of the robotic system of fig. 3.DETAILED DESCRIPTION

[0011] Described herein are examples of systems and processes for deployment of a robot in a work cell and for programming the work cell and / or the robot.

[0012] Fig. 1 show an example robotic system ("system") 100 with which the systems and processes described herein may be implemented. System 100 includes robotic arm ("arm") 101. Arm 101 includes robot joints ("joints") 102a, 102b, 102c, 102d, 102e, and 102f connecting a robot base ("base") 103 and a robot tool flange ("tool flange") 104.

[0013] In this example, example arm 101 includes six joints that are movable or rotatable; however, other implementations of arm 101 may include fewer than six joints that are movable or rotatable or more than six joints that are movable or rotatable. Arm 101 is thus a six-axis robot arm having six degrees of freedom enabled by the six joints. The joints in this example include the following: base joint 102a configured to rotate around base axis 105a; shoulder joint 102b configured to rotate around shoulder axis 105b; elbow joint 102c configured to rotate around elbow axis 105c; first wrist joint 102d configured to rotate around first wrist axis 105d; and second wrist joint 102e configured to rotate around second wrist axis 105e. As noted, the joints in this example also include joint 102f. Joint 102f is a tool joint comprising tool flange 104 configured to rotate around tool axis 105f. Robotic arm 101 also includes links 106a, 106b and 107. Link 106a is a cylindrical device that connects joint 102b to 102c. Link 107 is a 90° torus segment like angled device connecting joint 102c to link 106b. Link 106b is a cylindrical device that connects link 107 to joint 102d. It is noted that the links may have other shapes. Other implementations may include more or fewer joints. For instance, link 107 maybe provided as a second elbow robot joint rotating link 106b around a second elbow axis perpendicular with elbow axis 105c and first wrist axis 105d.

[0014] In this example, tool flange 104 is on an opposite end of arm 101 from base 103; however, that need not be the case in all robots. Tool flange 104 contains an end effector interface. The end effector interface enables an end effector to connect to arm 101 mechanically and / or electrically. To this end, the end effector interface includes a configuration of mechanical and / or electrical contacts and / or connection points to which an end effector may mate and thereby attach to arm 101.

[0015] In this example, the robotic system also comprises an end effector 108 (illustrated in dotted lines) in the form of a mechanical gripper attached to the robot tool flange. However, it is to be understood that the end effector can be any kind of end effector. The end effector may be any kind of passive tool that can be used by the robotic arm to perform a task and / or an active tool that also can perform actions independently of the operation of the robotic arm. Example of end effectors may be mechanical grippers, vacuum grippers, magnetic grippers, screwing machines, reverse screwing machines, welding equipment, gluing equipment, liquid or solid dispensing systems, painting equipment, visual systems, cameras, scanners, polishing equipment, nailing equipment, polishing equipment, wire holders, tubing holders, belt feeders, polishing equipment, force / torque sensors, distance sensors, laser-based tools, and / or others not listed here.

[0016] The robotic system can also comprise additional accessories attached to, mounted to or integrated into other parts of the robotic arm, for instance in form of wire holders, tubing holders, cameras, scanners, light indicators, distance sensors e.g. in form of safety skins, protective covers and / or other components not listed here.

[0017] In this example, the robotic system also comprises an additional component in form of a table 109, whereon workpieces can be placed. However, the robotic system can comprise any number of additional components and the additional components can be any kind of components, elements, or devices that constitute part of the robotic system and thereby form part of a work cell designed to carry out an automated work. Example of additional componentsmay be any kind of process equipment like conveyers, pallets, CNC machines, sorting machines, fixtures for arranging objects like workpieces, various safety equipment, visual systems, external axis, rotary tables, safety equipment etc. The additional components may also be mounting elements suitable for mounting elements / components of the robotic system, tables, pallets, stands, fences etc.

[0018] Robotic arm 101 includes one or more motors and / or actuators (not shown) associated with the tool flange and each joint. The one or more motors or actuators are responsive to control signals that control the amount of torque provided to the joints by the motors and / or actuators to cause movement, such as rotation, of the tool flange and joints, and thus of arm 101. For example, the motors and / or actuators may be configured and controlled to apply torque to one or more of the joints to control movement of the joints and / or links in order to move the robot tool flange 104 to a particular pose or location in the environment. In some implementations, the motors and / or actuators are connected to the joints and / or the tool flange via one or more gears and the torque applied is based on the gear ratio.

[0019] As also shown in fig. 1, robotic system 100 includes robot controller ("controller") 110 to control operation of arm 101. Controller 110 may be configured to output the control signals described herein to control movement, or restrain movement, of arm 101. Controller 110 may include, for example, one or more microcontrollers, one or more microprocessors, programmable logic such as a field programmable gate array (FPGA), one or more applicationspecific integrated circuits (ASICs), solid state circuitry, or any appropriate combination of two or more of these types of processing devices. In some implementations, controller 110 may include local components integrated into, or at a same site as, arm 101. In some implementations, controller 110 may include remote components that are remote in the sense that they are not located on, or at a same site as, arm 101. In some implementations, controller 110 may include computing resources distributed across a centralized or cloud computing service, at least a portion of which is remote from robotic arm 101 and / or at least part of which is local. The local components may receiveinstructions to control arm 101 from the remote or distributed components and control the motors and / or actuators accordingly.

[0020] Controller 110 may be configured to control motion of arm 101 by sending control signals to the motors and / or actuators to control the amount of torque provided by the motors and / or actuators to the joints. The control signals may be based on a dynamic model of robotic arm 101, a direction of gravity, signals from sensors (not shown) connected to or associated with each or some of the joints and / or links in the robotic arm, user-applied force, and / or a computer program stored in a memory 111 of controller 110. In this regard, the torque output of a motor is the amount of rotational force that the motor develops. The dynamic model may be stored in memory 111 of controller 110 or remotely and may define a relationship between forces acting on arm 101 and the velocity, acceleration, or other movement, or lack of movement of arm 101 that result(s) from those forces. In some implementations, memory 111 may include local components integrated into, or at a same site as, arm 101 or controller and in some implementations, memory 111 may include remote components that are remote in the sense that they are not located on, or at a same site as, arm 101 or controller 111.

[0021] The dynamic model may include a kinematic model of arm 101, knowledge about inertia of arm 101 and other operational parameters influencing the movements of arm 101. The kinematic model may define a relationship between the different parts / components of arm 101 and may include information about arm 101 such as the lengths and / or sizes of the joints and links. The kinematic model may be described by Denavit-Hartenberg parameters or like. The dynamic model may make it possible for controller 110 to determine which torques and / or forces that the motors and / or actuators should provide in order to move joints or other parts of the robotic arm, e.g., at a specified velocity, at a specified, acceleration, or to hold the robot arm in a static pose in the presence or absence of force(s).

[0022] Controller 110 may also include, or connect to, an interface device 112. Interface device 112 is configured to enable a user to control and / or to program operations of arm 101 via controller 110. Interface device 112 may be a dedicated device, such as a robotic teach pendent, which is configured tocommunicate with controller 110 via wired and / or wireless communication protocols. Such an interface device 112 may include a display 113 and one or more types of input devices 114 such as buttons, sliders, touchpads, joysticks, track balls, gesture recognition devices, keyboards, microphones, and the like. Display 113 may be or include a touch screen acting both as display and input device or user interface. Interface device 112 device may be or include a generic computing device (not shown), such as a smartphone, a tablet, or a personal computer including a laptop computer, configured with appropriate programming to communicate with controller 110.

[0023] The robotic system comprises a sensor system 118 configured to obtain sensor data indicative of one or more properties of the real environment of the robotic system. The real environment can be any part of the robotic system and thus a physical environment with physical objects. For instance, the real environment may be the surroundings around the robotic arm such as the surroundings of the robot base; parts of the robot system not including the robotic arm; parts of the robotic system before the robotic arm is arranged at the robotic system such as parts of the robotic system where the robotic arm is intended to be arranged or any other parts of the robotic system. The sensor system may comprise a number of sensing devices capable of obtaining sensor data indicating physical properties of the real environment such as visual properties like color, brightness and contrast information, structural information such as surface structures, temperature information indicating the temperature of parts of the real environment, acoustic information indicating acoustic properties of the real environment, etc. Examples of sensing devices may be visual cameras, stereo cameras, depth sensors, thermal sensors, acoustic sensors, etc.

[0024] In an example embodiment, the sensor system comprising some or more visual sensors of the same or different types(s), such as one or more three-dimensional (3D) cameras, one or more two-dimensional (2D) cameras, and / or one or more scanners, such as one or more light detection and ranging (LIDAR.) scanner(s). In this regard, a 3D camera is also referred to as an RGBD camera, where R is for red, G is for green, B is for blue, and D is for depth. The2D or 3D camera may be configured to capture sensor data in form of video, still images, or both video and still images. In some implementations, the image can be in form of visual information, depth information and / or a combination thereof, where visual information is indicative of visual properties of the environment such as color information and grayscale information, and the depth information is indicative of the 3D depth of the environment in the form of point clouds, depth maps, heat maps indicative of depth, or combinations thereof.

[0025] The sensor system 118 is configured to capture sensor data in a Field-Of-View (FOV). This FOV 119 may be based, at least in part, on the orientation of the sensor system in relation to the component(s) of the robotic system and directed towards the desired part of the real environment. In the example of fig. 1, the sensor system 118 is statically arranged in relation to robotic arm 101 and comprise a sensor device having a FOV with center at arrow 120 directed toward the robotic arm. The FOV of the sensor system may extend 10°, 20°, 30°, 40°, 50°, 60°, 70°, 80°, 90°, or more equally or asymmetrically on both sides of the center 120 and may increase with distance. The sensor system may also be configured as a movable sensor system that can change direction of the FOV for instance by arranging the sensor system on movable components. Also, the sensor system can be arranged on the robotic arm and move along with movement of the robotic arm. In some implementations, the vision system is fixed in the environment of the robotic arm, meaning that the vision system is not on the robotic arm and that its Field-Of-View is fixed in relation to the environment and does not move along with movements of the robotic arm. For instance, the vision system can be fixed to a monitor in a specified area of the environment around the robot base or integrated into an Augmented Reality device 115, as described hereafter.

[0026] The sensor system may be configured to obtain the sensor data in electronic form e.g. as analog or digital signals and to provide the sensor data to one or more processing and / or memory devices for processing and / or storage, for instance a processing unit and memory of a robot controller, a processing unit and a memory of an Augmented Reality devices or other processors and memories of the robotic system.

[0027] The robotic system comprises an Augmented Reality device (AR device) in form of a hardware and software system that provides a digital representation of the real environment based of its' position and orientation within the real environment using a display device, for instance the AR device may be configured to establish a virtual environment based on the sensor data and display the virtual environment on a display based on the position and orientation of the AR device. The virtual environment displayed on the display thus constitutes a digital representation of the real environment. The AR device may also be configured to overlay digital information onto the real environment of the robotic system or a representation of the real environment of the robotic system using a display device for instance by superposing the virtual environment onto a real environment or a representation of the real environment. The user's perception and understanding of the robotic system can hereby be enhanced. The AR device comprises at least one display device, sensors, and processors, where the display device presents digital content to the user, which is generated based on the position and orientation of the AR device within the real environment. For instance, the display device may be provided as a screen where digital content is displayed onto the display based on the position and orientation of the AR device in the real environment, such that the displayed digital content changes when the position and / or orientation of the AR device changes. In an embodiment the digital content may be superimposed on the user's view of the real environment of the robotic system. For instance, through various technologies such as head-mounted displays, handheld devices, or projection systems. In one embodiment a camera provides a (live) recording of the robotic system and displays the recording to the screen, and a video controller can then overlay the digital content in form of graphical elements with the video recording of the robotic system. The display device may be provided as a transparent screen where the user can observe the robotic system through the transparent screen and where the transparent screen can display information that thereby are overlayed the user's view of the robotic system. Further, the display device may also be provided as a projecting device e.g., a video projector or the like that can project images or the like onto the real robotic system, whereby graphical content is overlayed onto the realenvironment of the robotic system. The sensors may capture data about the user's environment and their interaction with it. This may include cameras for visual data, microphones for audio data, and motion sensors for tracking the AR. device's movements. The processors interpret the sensor data and generate the digital content to be displayed. This involves algorithms for understanding the environment, tracking the AR device's position and orientation, and rendering the digital content in real-time. The AR device may also include input mechanisms for the user to interact with the digital content, such as touchscreens, gesture recognition, or voice commands.

[0028] In this example, the AR device 115 is configured to establish the relative position (illustrated by position vector 117a and orientation vector 117b) of the AR device in relation to the robotic arm. This can be done by utilizing any kind of position and orientation determining systems, techniques and methods as known from the field of Augmented Reality systems. For instance, the AR device may be equipped with sensors such as accelerometers, gyroscopes, and magnetometers and can use these to measure movement and orientation, estimating the device's position and orientation relative to an original known position and orientation. The original known position and orientation can for instance be obtained by arranging the AR device at a predefined position and orientation or by using a camera to capture images of the environment. Distinctive features in these images can be identified and tracked across multiple images to estimate the device's movement relative to the environment. For instance, the AR device can use information of the robotic arm and its dimensions to, from a picture or video recording the robot, establish a spatial reference system, such as the robot base coordinate system i.e. of the three coordinate axes Xbase, ybase, Zbase (an example of a spatial reference system). The robotic arm can also be equipped with visual marks that the AR device can use to determine its position and orientation in relation to the robotic arm. Other position and orientation determining techniques such as map-based positioning, global positioning systems, network-based positioning, LiDAR-based positioning, radar-based positioning, acoustic-based positioning, visual-based positioning (image based) like visual odometry, inertial-based positioning using IMU (Inertial measurement unit) or combinations thereof may also be used.

[0029] The AR. device is configured to establish a virtual environment based on sensor data obtained by the sensing system 118 of the robotic system. The virtual environment is a digital representation of at least some properties of a part of the real environment of the robotic system and can for instance include virtual objects that correspond to real-world objects where the virtual objects have properties that are defined based on the sensor data. For example, the color of a virtual object is based on the color data captured by the color sensor, the brightness of the virtual environment is based on the brightness data captured by the light sensor, the texture of a virtual object is based on the surface structure data captured by the texture sensor, and the temperature of a virtual object is based on the temperature data captured by the temperature sensor etc. The virtual environment can for instance be provided as a 3D world space as known from 3D visualization systems. The 3D world space may for instance comprise virtual surface elements representing surface structures of the robotic system.

[0030] The AR device is configured to provide a digital representation of the real environment based on its' position within the real environment using a display device. As described above this can be achieved through various AR technologies such as head-mounted displays, handheld devices, or projection systems. In the illustrated example, the AR device 115 is a portable AR device 115 having a physical display 116 and a camera (not shown), where the camera captures a representation of the real environment of the robotic system in form of a video feed displayed at the physical display. In the illustrated example the physical display shows a video representation of the table 109' and the end effector 108'. The AR device superposes a visual representation of the virtual environment onto the video feed as graphical elements based on the position vector 117 of the AR device in relation to the robotic arm . Consequently, the AR device can align the virtual environment with the real environment shown at the physical screen and graphical elements of the virtual environment can hereby be superpose onto specific elements of the real environment. In this example the virtual environment is illustrated as four spheres 121 added onto of the video feed at corners of the table 109'. If a user moves the AR devise around inthe environment of the robotic system, then the camera will record other parts of the robotic environment and the AR. device can be configured to adjust the superposed visual environment according to the movement of the AR device, thus the virtual environment on the display appears as elements forming part of the robotic environment.

[0031] The AR device is configured to receive inputs from a user enabling the user to select parts of the virtual environment, for instant via any kind of user input devices such as touch screens, buttons, sliders, touchpads, joysticks, track balls, pointing devices, gesture recognition devices, keyboards, microphones, and the like. In this example, the physical display 116 is provided as a touch screen and the user can select elements of the virtual environment by touching the relevant parts displayed at the physical display. For instance, in fig. 1, a user selects one of the spheres 121 by touching the sphere at the screen as illustrated by touching finger 122. The AR device can also be configured to provide feedback to the user when selecting the parts of the virtual environment for instance in the form of an indication that a part of the virtual environment has been selected. The feedback can for instance be in form of visual, acoustic and / or haptic feedback. For instance, the color of the virtual element may change upon selection by the user, the AR device may play a sound upon selection, the AR device may vibrate upon selection. The AR device can also be configured to enable the user to deselect selected parts, for instance via any kind of input devices, this enables the user to deselect parts of the virtual environment which have unintentionally been selected.

[0032] In an embodiment, the AR device comprises an artificial 3D selection tool configured to select parts of the virtual environment based on the relative position and / or orientation of the AR device in relation to the spatial reference system of the robotic system and based on a volume defined in relation to the AR device. This makes it possible to use the AR device as a selection tool which can be moved around in the real environment of the robotic system and where corresponding virtual elements of the virtual environment is selected when the AR device is positioned near the corresponding real elements.For instance, the AR. device may be configured to select the virtual elements of the virtual environment when the corresponding real elements are within a predefined volume defined in relation to the AR device. The predefined volume may for instance be any kind of volume surrounding the AR device such as spheres, cuboids pyramids, cones, polyhedrons etc. Similar, the AR device can also comprise an artificial 3D deselection tool configured to deselect parts of the virtual environment based on the relative position and / or orientation of the AR device in relation to the spatial reference system.

[0033] In one embodiment the volume is defined as a cuboid surrounding the AR device, and the AR device can be switch into a selection mode. In the selection mode the AR device is configured to select the virtual elements corresponding to the real elements when the real objects are within the cuboid. This makes it possible for the user to select virtual elements by moving the AR device around in the real environment and close to the real elements that needs to be selected. The AR device can in such an embodiment be used as an artificial painting sponge, where the AR device virtually paints the part of the virtual environment when it is move close to the real elements and the virtually painted parts can be selected. It is noted that the same principle also can be used when a user wants to deselect already selected parts, for instance the AR device can be used as an artificial eraser where the AR device virtually erases the paint of the virtually painted parts of the virtual environment when it is moved close to the real elements. The virtually erased parts can in this way be deselected.

[0034] In one embodiment the volume is defined as a cone having its apex at a surface of the AR device and where the cone protrudes from the surface. In the selection mode the AR device is configured to selects the virtual elements corresponding to the real elements when the real objects are within the cone. This makes it possible for the user to select virtual elements by moving the AR device around in the real environment and close to the real elements that needs to be selected. The AR device can in such an embodiment be used as an artificial spray paint, where the AR device virtually spray paints the part of the virtual environment that are within the cone. In an embodiment where the AR device is provided as a handheld device with a camera the cone may be arranged with its apex at the camera whereby the user can consider the camera as equivalentto the nozzle of a paint can. It is noted that the same principle also can be used when a user wants to deselect already selected parts, for instance the AR. device can be used as an artificial spray paint eraser where the AR device virtually erases the paint of the virtually painted parts of the virtual environment when it is moved close to the real elements. The virtually erased parts can in this way be deselected.

[0035] In one embodiment the AR device is provided with a user interface enabling the user to activate and deactivate the selection / deselection tool. This enables to user to activate the selection / deselection tool when the AR device near the elements that needs to be selected and / or deselected. The user can then de-activate to selection / deselection tool when the AR device needs to be moved around in the real environment without selecting / deselecting parts of the virtual environment.

[0036] In one embodiment the AR device comprises a virtual 3D selecting tool moveable by a user within the virtual environment, where the virtual 3D selecting tool is configured to select virtual elements within a predefined 3D virtual volume. The virtual 3D selecting tool can for instance be displayed as part of the displayed virtual environment where the user can move the virtual 3D selecting tool around for instance via a touchscreen. The virtual 3D selecting tool can for instance be provide as a cuboid shape and be configured to act as a virtual painting sponge or as a cone configured to act as a virtual spray paint in similar ways as previously described in connection with the artificial painting sponge and artificial spray paint.

[0037] The AR device is configured to enable a user to associate context to the selected parts of the virtual environment. This can for instance be achieved via a user input device enabling the user to select one of more contexts that shall be associated with the selected parts. In one embodiment, the AR device is configured to enable the user to select one or more contexts from a set of predefined contexts displayed to the user via a graphical user interface. The AR device can also be configured to enable the user to select one or more contexts that shall be associated with selected parts of the virtual environment subsequentially selected by the user. For instance, the user may select one ormore predefined context(s) that shall be associated with parts of the virtual environment that the user subsequentially selects. It is thus to be understood that the order of selecting and associating context to the selected parts can mixed as desired.

[0038] The selected parts of the virtual environment and the associated context can for instance be stored in a database associated with the robotic system, where the database provides a link between the selected parts of the virtual environment and the associated context. The database can for instance be stored in a memory of the AR. device, the robotic system and / or an external memory device.

[0039] The context associated with the parts of the virtual environment refer to the circumstances in which the selected parts form part of the robotic system, and can for instance indicate states, properties, functions of the selected parts; the relationship that the selected parts have to other parts of the robotic system such as the relationship to other devices or humans in the robotic system. The context of the selected parts may have an impact on the performance and behavior of itself and / or the robotic system, and the context it is often considered when designing and programming robotic systems.

[0040] The robot controller 110 is configured to control the robotic system based on the selected parts of the virtual environment and the context associated with the selected parts. The robot controller can hereby utilize the associated context of the parts of the virtual environment when controlling the robot system, and a user programming and setting up the robot system can thus easily provide information about the intended operation of the robot system to the robot controller.

[0041] The robot system according to the present invention makes it possible for a user to model the properties of the robot system by associating context to parts of the robot system which can be used by the robot controller during control of the robotic system. The deployment time of the robotic system can hereby be reduced significantly as the context associated to parts of the robot environment can be used to provide instructions to the robot controller.

[0042] The context associated with the selected parts of the virtual environment can for instance define interact properties indicating structures of the robotic system, that for instance the robotic arm and / or an end effector of the robotic arm can interact with. The robot controller can then be configured to control the robotic arm and / or the end effector by executing a robot program causing the robotic arm and / or end effector to interact with parts of the virtual environment based on the interact properties. This can for instance be workpieces that the robotic arm can manipulate, for instance the interact property can indicate areas of the workpiece that is suitable for being gripped by a gripper of the robotic arm; indicate workpieces or parts of the workpieces to which the robotic arm and / or an end effector should provide some kind of treatment, such as grinding regions, polishing regions, milling regions, painting regions etc. that the robot arm and / or end effector respectively need to grind, polish, mill, paint etc. The interact property can also be contact points like buttons, doors that the robotic arm need to interact with. For instance, in the field of machine tending, the robotic arm may during the operation need to open and close a door of a CNC machine and the indication property may in such situation indicate the handle of the door as an interaction point.

[0043] The context associated with the selected parts of the virtual environment can for instance define avoid properties indicating structures of the robotic system that the robotic arm and / or an end effector of the robotic arm must avoid during movement of the robotic arm. This can for instance be the surface structure of other machinery forming part of the robotic system such as conveyor belts, feeders, CNC machines, parts of the workpieces etc. Also, the avoid properties can indicate boundaries of openings such as the edges of a door opening of an CNC machine, the edge of a box from which the robotic arm and / or end effector shall pick workpieces. The robot controller can then be configured to control the robotic arm and / or the end effector by executing a robot program causing the robotic arm to move around the parts associated with avoid properties. For instance, the robot controller can be configured to execute a path planning algorithm which takes into account the geometric position of the selected parts associated with avoid properties and thereby arriveat a movement where the robotic arm and / or end effector avoid certain parts of the robotic system.

[0044] The context associated with the selected parts of the virtual environment can for instance define ignore properties indicating structures of the robotic system that the robotic arm and / or an end effector of the robotic arm can ignore during movement of the robotic arm. This can for instance be the surface structure of elements or components forming part of the robotic system during the examination, integration, configuration, and / or test phase and which are removed before the robotic system is put into operation. The robot controller can then be configured to control the movements of the robotic arm while ignoring the parts associated with ignore properties. For instance, the robot controller can be configured to execute a path planning algorithm which ignores the selected parts associated with ignore properties.

[0045] In an embodiment, the context associated with the selected parts of the virtual environment can define one or more state properties defining a state of the selected parts of the virtual environment. The state property can indicate a state of a real object of the robotic system represented by the selected parts of the virtual environment and can be a static state that does not change during operation of the robotic system or a dynamic state that can change during operation of the robotic system. The state property can be used by the robot controller during operation of the robotic system for instance as parameters in the robot program. The state property can for instance be associated with one or more I / O ports of the robot controller. As an example, the selected parts of the virtual environment may represent a sensor sensing if an object is present at a pickup position at the end of a conveyer. The state property may then be associated with the output of the sensor and thereby indicate the presence of the object at the pickup position. The user can for instance select the sensor and indicate the state property as 1 when the object is present and 0 when the object is not present and associate these values with an I / O port of the robot controller. The robot controller can then read the value of the I / O port and initiate pickup of the object when the value of the I / O is 1. In another example, the user can select a part of the virtual environment representing a door of a CNC machine and associate the CNC machine's dooroutput line with an I / O port of the robot controller and indicate the state property as 1 when the door is open and 0 when the door is closed. The robot controller can then read the value of the I / O port and let the robotic arm interact with the CNC machine based on the state of the CNC door.

[0046] In an embodiment, the context of selected parts of the virtual environment can define one or more action properties defining actions that can be performed by the selected parts. The action properties may for instance define actions that can be performed by devices of the robotic system. For instance, a CNC machine may be associated with a processing action defining the processing that a CNC machine carries out on a workpiece associated with a processing action that processes the workpieces, a metal pressing machine may be associated with a metal pressing process that can be carried out by the metal press; a conveyer may be associated with a conveyer actions indicating movements of a conveyer; a measuring device may for instance be associated by a measuring action defining a measuring process carried out by the measuring device; etc.

[0047] In an embodiment, the context associated with the selected parts of the virtual environment can define one or more geometric properties relating to geometric properties of at least a part of an object of the robotic system . The geometric properties can for instance be in form of basic 2D primitives such as triangle, circle, rectangle, etc. on one of the faces of the object. However, the geometric properties may also be in form of 3D primitives such as spheres, cuboids, cubes, cylinders, cones, pyramids, polyhedrons etc.

[0048] The present invention also relates to a method of deploying a robotic system. In general, deployment of a robot system refers to the process of integrating a robotic system into an operational environment. The method may comprise different phases such as, an examination phase comprising examining (manual) processes and using them as a starting point for planning the robotic system, an integration phase comprising integrating the robotic system by bringing together the components of the robotic system, a configuration phase comprising configuring the components into a functional robotic system, a testing phase comprising testing operation of the roboticsystem in order to ensure that the processes carried out by the robotic system functions as intended and an operation phase operating the robotic system to carry out the intended process. It is to be understood that the various phases are not strictly separated and that in many robot deployments the phases overlap.

[0049] Fig. 2 illustrates an embodiment of a method 229 of deploying a robotic system according to the present invention. The method comprises a step 230 of providing a robotic arm and at least one additional component; a step 235 of obtaining sensor data indicating properties of the real environment of the robotic system; a step 240 of establishing a virtual environment of the robotic system; a step 250 of providing at least one AR. (Augmented Reality) device, a step 260 of selecting parts of the virtual environment using the AR device, a step of 270 of associating context to the parts of the virtual environment and a step 280 of deploying the robotic system based on the context associated with the selected parts of the virtual environment.

[0050] Step 230 of providing a robotic arm and an additional component can be performed by identifying a desired robotic arm and other components necessary for the desired robotic application. The robotic arm can be any kind of robotic arm and the components can be any kind of components, elements, devices, sensors etc. intended to form part of the robotic system. For instance, the robotic arm and the additional component can be any of the robotic arms and components previously described in connection with fig. 1. In one embodiment, the step 230 of providing the robotic arm and the additional component comprises a step (not shown in fig. 2) of arranging at least the robotic arm and / or the at least one additional component of the robotic system at a desired position. For instance, in an embodiment the robotic system may comprise a plurality of additional components and these may be arranged at their desired positions without arranging the robot arm at the desired position. This makes it possible to perform the method to the present invention prior to arranging the robotic arm at the robotic system and then utilizing the results of the method when arranging the robotic arm. However, it is to be understood that the step of providing the robotic arm and the additional component alsomay comprise a step of arranging the robotic arm at a desired position of the robotic system and utilize the results of the method to arrange other components of the robotic system.

[0051] Step 235 of obtaining sensor data indicating properties of the real environment of the robotic system can be performed by utilizing a sensor system configured to obtain the sensor data. The step may for instance comprise steps of activating the sensor system and storing the obtained sensor data in a memory of a computer system. It is also to be understood that the step of obtaining the sensor data may comprise a step of processing the obtained sensor data such as managing, analyzing, and / or manipulating the sensor data using computational methods. The purpose of the data processing of the sensor data is to prepare to sensor data for establishing a virtual environment. The real environment can be any part of the robot system and thus a physical environment with physical objects. For instance, the real environment may be the surroundings around the robotic arm such as the surroundings of the robot base, parts of the robot system not including the robotic arm, parts of the robotic system before the robotic arm is arranged at the robotic system, such as parts of the robotic system where the robotic arm is intended to be arranged or any other parts of the robotic system. The sensor system may comprise a number of sensing devices capable of obtaining sensor data indicating physical properties of the real environment such as visual properties like color, brightness and contrast information, structural information such as surface structures, temperature information indicating temperature of parts of the real environment, acoustic information indicating acoustic properties of the real environment, etc. Examples of sensing devices may be visual cameras, stereo cameras, depth sensors, thermal sensors, acoustic sensors, etc. In an example embodiment, the sensor system comprising some or more visual sensors of the same or different types(s), such as one or more three-dimensional (3D) cameras, one or more two-dimensional (2D) cameras, and / or one or more scanners, such as one or more light detection and ranging (LIDAR.) scanner(s). In this regard, a 3D camera is also referred to as an RGBD camera, where R is for red, G is for green, B is for blue, and D is for depth. The 2D or 3D camera may be configured to capture sensor data in form of video, still images, or bothvideo and still images. In some implementations, the image can be in form of visual information, depth information and / or a combination thereof, where visual information is indicative of visual properties of the environment such as color information and grayscale information, and the depth information is indicative of the 3D depth of the environment in the form of point clouds, depth maps, heat maps indicative of depth, or combinations thereof.

[0052] Step 240 of establishing a virtual environment of the robotic system based on the sensor data can be performed by utilizing a computing device configured to establish the virtual environment based on the sensor data obtained by a sensing system. The virtual environment is a digital representation of at least some properties of a part of the real environment of the robotic system and can for instance include virtual objects that correspond to real-world objects where the virtual objects have properties that are defined based on the sensor data. For example, the color of a virtual object is based on the color data captured by the color sensor, the brightness of the virtual environment is based on the brightness data captured by the light sensor, the texture of a virtual object is based on the surface structure data captured by the texture sensor, and the temperature of a virtual object is based on the temperature data captured by the temperature sensor etc. The step 240 can thus be carried out via a 3D virtual space as known from 3D visualization systems.

[0053] Step 250 of providing at least one AR. (Augmented Reality) device can be performed by providing an AR device as described previously in connection with fig. 1. The step of providing the AR device comprises a step 251 of establishing the relative position and orientation of the AR device in relation to a spatial reference system of the robotic system, which may be carried out by utilizing any kind of positioning determine systems, techniques and methods known from the field of Augmented Reality. Once the position and orientation of the AR device in relation to the spatial reference system of the robotic system has been established, a step 252 of displaying the virtual environment on a display based on the position and orientation of the AR device is performed. Step 250 can also comprise an optional step 253 of superposing the virtual environment onto the real environment of the robotic system or arepresentation of the real environment of the robotic system. The step 253 can be performed by using the AR. device to project the virtual environment onto the real environment of the robotic system utilizing a projecting system, to overlay the virtual environment to a user's field of view by utilizing transparent screens though which the user observes the robotic system, or to overlay the virtual environment onto a recording of the robotic system and display the recording with the virtual environment on a screen.

[0054] Step 260 of selecting parts of the virtual environment via the AR device can be performed by utilizing a user input devise configured to receive inputs from a user by enabling the user to select parts of the virtual environment; for instance, via any kind of user input devices such as touch screen, buttons, sliders, touchpads, joysticks, track balls, pointing devices, gesture recognition devices, keyboards, microphones, and the like. The method can also comprise an optional step of deselecting selected parts via the AR device. This enables the user to deselect parts of the virtual environment that have unintentionally been selected.

[0055] Step 270 of associating context to parts of the virtual environment can be performed by utilizing user input devices configured to receive inputs from a user. The user input device can enable the user to select one of more contexts that shall be associated with parts of the virtual environment. In one embodiment, the user can select one or more contexts from a set of predefined contexts displayed to the user via a graphical user interface of the AR device. In the illustrated embodiment, the selected context is associated to the parts of the virtual environment selected in step 260. However, the order of step 260 and step 270 can be reversed, such that the user first selects the context to be associated with parts of the virtual environment and then associate the selected context to parts of the virtual environment by selecting parts of the virtual environment. Step 270 can also comprise associating more than one context to the same parts of the virtual environment. The method can also comprise an optional step of de-associating context associated with parts of the virtual environment. This makes it possible to de-associate context of parts of the virtual environment that previously unintentionally have been associated with a context. The context associated with the parts of the virtual environment referto the circumstances in which the selected parts from part of the robotic system for instance states, properties, functions of the selected parts; the relationship that the selected parts have to other parts of the robotics system such that the relationship other devices or humans in the robotic system. The context of the selected parts may have an impact on the performance and behavior of itself and / or the robotic system, and the context is considered when designing and programming robotic systems. Further non-limiting examples of context have been described previously.

[0056] Step 280 of deploying the robotic system based on the context associated with parts of the virtual environment can be performed by using the associated context of parts of the virtual environment in any of the general phases (described in paragraph [00481F-00 7--1-) of robot system deployment, such as the examining phase, the integration phase, the configuration phase, the testing phase and / or the operation phase. Deploying the robot system based on context associated with part of the virtual environment, where the context is associated with the virtual environment by utilizing an AR. device makes it possible to in an easy and intuitive way to deploy the robotic system, as in most robotic systems a user needs to define the context in which parts of the robotic system are intended to work. The virtual environments represent the physical environment of the robotics system and by associating context to the virtual environments representing the real environment makes it easier to deploy the robotic system as the context associated to the virtual environment can easily be assigned to corresponding parts of the real environment and various software can then be utilized during the deployment. That the virtual environment displayed on the AR device based on the position and orientation of the AR device in the robotic system makes it possible for the user to move the AR device around in the real robotic environment and thereby arrange the AR device at positions such that the relevant parts of the robotic system is displayed on the AR device. The user can hereby easily move the AR device around the real environment and associate context to the parts for the robotic system via the AR device.

[0057] The following paragraphs provide a number of non-limiting examples of how method of deploying step 280 of deploying the robotic systembased on the context associated with parts of the virtual environment can be performed in different phases of the robot deployment.

[0058] In the examination phase of a deployment of a robotic system, the virtual environment can be obtained based on the sensor data of the real environment of a working station which is being examined for the possibility of utilizing a robotic arm to carry out a working process. The context associated to parts of the virtual environment can be points of interest, such as positions where the robotic arm shall carry out a certain task, such as pick up a workpiece, place a workpiece, screwing, milling etc. and the step of deploying the robotic system can comprise a step of determining what kind of a robotic arm is suitable for the working process. This can for instance be carried out by comparing the distance between the points of interests of the virtual environment with the reach of a number of predefined robotic arms and then based on the comparison identifying if any of the robotic arms are suitable for carrying out the working process. Such comparison may for instance be implemented into a software solution that identifies the distance between the points of interest indicated by the context associated with the parts of the virtual environment and then compare that distance with a database comprising information in relation to the reachability of the robotic arms.

[0059] In the integration phase of a deployment of a robotic system, the virtual environment can be obtained based on the sensor data of the real environment of the working station into which the robotic system is being integrated. The context associated to parts of the virtual environment can be points of interest, for instance positions where the robotic arm shall carryout a certain task, such as picking up a workpiece, placing a workpiece, screwing, milling etc. and a placement area where the robotic arm can be arranged. The step of deploying the robotic system can comprise a step of determining the optimal position of the robotic arm within the placement area and thereafter arranging the robotic arm at the determined position. This can for instance be done by performing an optimization process based on the context of the virtual environment and the properties of the robotic arm, for instance the optimization process may be implemented as a software solution that identifies the distance between the points of interest indicated by the context associated with parts ofthe virtual environment and then compare that distance with properties of the robotic arm and use the context indicating the desired placement area of the virtual environment as a solution space.

[0060] In the configuration phase of a deployment of a robotic system, the virtual environment can be obtained based on the sensor data of the real environment of the robotic system after the robotic arm has been integrated into the robotic system. The context associated to parts of the virtual environment can be points of interest such as positions where the robotic arm shall carry out a certain task, such as picking up a workpiece, placing a workpiece, screwing, milling, etc. The step of deploying the robotic system can comprise a step of providing robot code to control the robotic arm based on the context associated with the virtual environment. For instance, the context may indicate a pick position where a workpiece is to be picked up, a placement position where to the workpiece is to be delivered and avoid properties indicating elements of the virtual environment that must be avoided during movement of the workpieces. This can for instance be done by performing a path planning algorithm providing a robot trajectory moving the workpiece from the pickup position to the placement position without parts of the robotic arm or the workpiece colliding with the avoid properties. The path planning algorithm thus optimizes the path based on the context of the virtual environment, and since the virtual environment has been made based on sensor data indicating properties of the real environment, the determined path corresponds to a real path in the real robotic environment when executed as a robotic movement by the robot controller.

[0061] In the testing phase of deployment of a robotic system, the virtual environment can be obtained based on the sensor data of the real environment of the robotic system that needs to be tested. The context associated to parts of the virtual environment can indicate parts of the virtual environment that shall be tested, for instance test positions that the robotic arm shall visit in order to verify the functionality of the working process. The control system of the robotic system can then be configured to control the robotic arm to move to these positions. Also, during the testing phase, it is possible to perform test executions of the process carried out by the robotic system and the user canthen observe the test execution utilizing the AR. device and during the execution of the test associate context to the virtual environment. The context can then be used to correct the operation of the robotic arm. For instance, the context may indicate structures of the robotic system which the robotic arm must keep a predefined distance to avoid persons getting caught between the robotic arm and the structure. The user can for instance observe that the robotic arm moves too close to a structure and then associate context of the representation of the structure in the virtual environment, where after the robot control system can be configured to correct movement of the robotic arm accordingly.

[0062] In the operating phase of deployment of a robotic system, the virtual environment can be established based on the sensor data of the real environment of the robotic system in operation. The context can, for instance indicate a new pickup position indicated by the operator operating the robotic system via the AR device. The robot controller can be configured to adjust its robot program so that at the next iteration the pickup position is changed based on the context associated to the virtual environment by the operator.

[0063] Figs. 3-8 illustrate aspects an experiment performed by the inventors and serves to illustrate how the robotic system and the method according to the present invention can be used to deploy a robotic system. The experiment has been performed as an examination phase investigating the placement of a robotic arm in a robotic system where the robotic arm is intended to load workpieces into an CNC machine.

[0064] Fig. 3 illustrates a gray scale image of the experimental robotic system 300 used for illustrating and testing the robotic system and the method according to the present invention. The experimental robotic system comprises a first table 309a with a picking area 323 from where the robotic arm is supposed to pick a workpiece. The picking area 323 is marked with bright tape 324 appearing as a bright square in fig. 3. Detailed part A illustrates an enlarged view of the picking area 323 and the robotic arm is supposed to be arranged on the first table at a placement area 328 indicated near a corner of the first table. An imitated CNC machine is provided at a second table 309b where on a rackimitating the door 325 of a CNC machine is arranged. Detailed part B illustrates the imitated door 325 seen from the front. A box 326 is arranged at the second table and comprises a chuck 327 of a CNC machine and the workpiece needs to be loaded into the chuck. Detailed part C illustrates, that the chuck 327 is accessible through an opening in a side of the door 325. The robotic arm is to be arranged on the first table and be able to pick up a workpiece from the picking area and thereafter load the workpiece into the chuck without any parts of the robotic arm and workpieces colliding with the boundaries of the door 325.

[0065] In the experimental robotic system, the AR. device is provided as a smartphone (iPhone 13 Pro Max) comprising a LiDAR scanner, where the LiDAR scanner is used to obtain sensor data of the real environment, and the AR device comprises software establishing a virtual environment representing the real environment of the experimental robotic system. In the experimental robotic system, the virtual environment is provided as a point cloud comprising a plurality of cloud points indicating surface structure points of the experimental robotic system in a 3D world space. The AR device is configured to display the virtual environment on a screen based on the position and orientation of the AR device in relation to the experimental robotic system. In the illustrated embodiment, the 3D virtual environment in the form of the point cloud is displayed on the screen as the 3D virtual environment would be seen through the back camera of the AR device. Consequently, the view of the 3D environment changes with movements of the AR device and the user then observes the 3D environment on the screen from the same viewing angle as the position and orientation of the AR device.

[0066] The virtual environment in form of a point cloud is thus obtained based on a consumer-grade hardware - such as a mobile device equipped with a LiDAR scanner and a camera, which makes it possible to utilize general available hardware such as smart phones and tablets. However, it is to be understood that special hardware such as robot teach pendants provided with a LiDAR scanner also can be used as the AR device. In an embodiment, the point cloud is obtained by capturing an RGB image using the camera and a depth map using the LiDAR scanner. The sampled RGB information and depth values are then converted into points having RGB and a position in the 3D world. In theexperiment, the points of the point cloud are generated based on the position and orientation of the AR. device and a defined grid size of the image plane having a resolution of 2000 points. The RGB and depth values from the image plane are then projected to a 3D world space based on the camera's projection transformation. A new pair of RGB image and depth map is thereafter captured whenever the camera either changes orientation by 1° or moves by 0.01m and a new set of cloud points are obtained and added to existent point cloud data. It is possible to reduce the volume of data and only capture the data in the area of interest by requesting the user define a virtual bounding box of the robotic system whereby the AR device only will capture data within that bounding box.

[0067] Figs. 4a and 4b illustrate exemplary views of the virtual environment of the robotic system 300' of the experimental robotic system and corresponding component has been given the same reference numbers as in fig. 3, but in figs. 4a and 4b denoted with a prime sign ' to indicate that the components in figs. 4a and 4b are virtual representations of the real components. The virtual environment of the robotic system 300' is provided as a point cloud obtained by the AR device, where each point indicates a surface structure of the real environment of the experimental robotic system shown in fig. 3. Here the virtual environment is illustrated as a grayscale image provided based on the RGB information of the cloud points. However, it is to be understood than the cloud points also can indicate the color of the structure which can be displayed on a color display.

[0068] Fig. 4a illustrates the virtual environment of the robotic system 300' from a position above the first table and in front of the imitated door. The virtual environment of the robotic system 300' shows a virtual representation of the imitated door 325', a virtual representation of the box 326', a virtual representation of the chuck 327', a virtual representation of the first table 309a' and a virtual representation of the second table 309b'.

[0069] Fig. 4b illustrates the virtual environment of the robotic system 300' from a position besides the first table and facing towards the imitated door of the CNC machine. In this view, the virtual environment of the robotic system 300' shows a virtual representation of the imitated door 325', a virtualrepresentation of the box 326', a virtual representation of the second table 309b' and a virtual representation of the first table 309a' whereon the tape 324' indicating the pick area is also indicated.

[0070] Figs. 5a and 5b illustrate the exemplary views of the virtual environment of the robotic system 300' of figs. 4a and 4b, where a user has selected the rack imitating the door 325', a part of the second table 309b' and a part of the upper surface of the second table 309a'. The selected parts are illustrated as dark nearly black parts of the virtual environment. In the experimental robotic system, the user then associate context to the selected parts of the rack 325' and the second table 309b' as avoid properties 591 (some of the dark parts) indicating structures that the robotic arm and / or an end effector of the robotic arm must avoid during movement of the robotic arm, as the robotic arm must avoid colliding with these parts during operation. Further, the user associates the context to the selected part of the first table 309a' as optimize properties 592 indicating an area of the second table where the robotic arm can be arranged. The optimize properties may also comprise an indication of a mounting vector 593 indicating a desired mounting direction of the robot base, for instance in form of one or more mounting vectors 593 indicating the direction of one or more coordinate axes of a robot base coordinate system i.e. one or more the three coordinate axes Xbase, ybase, Zbase of the robot base illustrated in fig. 1.

[0071] Figs. 6a and 6b illustrate the exemplary views of the virtual environment of the robotic system 300' of figs. 4a and 4b, where a user has selected the box 326'. The selected parts of the box are illustrated as dark nearly black parts of the virtual environment. In the experimental robotic system, the user associates the context to the selected parts of the box as avoid properties 591 indicating structures that the robotic arm and / or an end effector of the robotic arm must avoid during movement of the robotic arm.

[0072] Figs. 7a and 7b illustrate the exemplary views of the virtual environment of the robotic system 300' of figs. 4a and 4b, where a user has selected the chuck 327' (visible in fig. 7a) and the picking area 323' at the first table 309a' (visible in fig. 7b). The selected parts are illustrated as dark nearly black parts of the virtual environment. In the experimental robotic system, theuser associates the context of the selected parts of the first table and the selected chuck as interact properties indicating that the robotic arm performs some kind of interaction with the selected parts. The interact properties of the selected parts of the first table are defined as pickup properties 594 indicating a pickup area where the robotic arm shall pick up a workpiece. The pickup propertied may also include information on how the workpiece can be picked up for instance in form of a pickup vector 595 indicating a path along which a gripper need to approach the workpieces to grip the workpiece. The pickup vector can also indicate how the gripper need to retract from the picking area with the workpiece. In the illustrated embodiment, the pickup vector is illustrated as a double arrow indicating that the approach path and the retraction path are the same. However, it is to be understood that the approach path and retraction path may be different. The interact properties of the selected parts of the chuck are defined as delivery properties 596 where the robotic arm needs to place the workpiece. The delivery properties may include information of how the workpiece needs to be inserted into the chuck and may for instance define a delivery vector 597 indicating a path along which a gripper need to insert the workpiece into the chuck. The delivery vector can also indicate how the gripper need to retract from the chuck after having delivered the workpiece into the chuck. In the illustrated embodiment the delivery vector is illustrated as a double arrow indicating that the insertion path and the retraction path are the same. However, it is to be understood that the insertion path and retraction path may be different.

[0073] In figs. 5a to 7b, the selected parts are illustrated as dark nearly black parts, however it is to be understood that the selected parts can be indicated in many ways, for instance by utilizing colors when the virtual environment is displayed on a color display. In one embodiment the RGB information of the selected points can for instance be color shifted towards a predefined color, such that the selected parts to some extend maintain original color which is toned. This enhances the user's understanding of the robotic system and makes it easier for the user to identify different selected parts. Alternatively, or in addition, the color of the selected parts can be determined based on the context associated with the selected parts. For instance, structuresassociated with avoid properties may be indicated in red or color shifted towards red, interact properties may be indicated in green, or color shifted towards green, optimize properties areas may be indicated in blue, or color shifted towards blue. It is to be understood that the suggested color scheme only serve as an example and that the skilled person may adjust the color scheme as desired. In the user interface of AR. device of the experimental robot system the avoid properties are indicated as red points, the pickup properties as blue points, the placement properties as green points and the optimize properties as white points.

[0074] Figs. 5a to 7b serve to illustrate how the user in the experimental robotic system can use the AR device to associate context to parts of the robotic system. The virtual environment of the robotic system with the associated context to parts of the robotic system is then used to deploy the robotic system by finding the optimal position of the robotic arm within the selected part of the first table 309a' associated with optimize properties. This can be achieved by executing an optimization process that can be carried out by the robot controller of the robotic system or another processing device.

[0075] In one embodiment the deployment process comprises a data processing process where the point clouds associated with context are processed to optimize the deployment process.

[0076] The point clouds associated with context may be optimized for the deployment process by removing outliers of a group of cloud points. Hereby it can be avoided that outliers (e.g., due to sensor errors) influence the deployment process. Hereby a group of cloud points associated with the same context can be reduced to include only inliers.

[0077] Point clouds associated with context may also be optimized by generating convex hulls based on a group of cloud points for instance by utilizing the method according to reference {1.}. The convex hulls can thereafter be used in the deployment process, for instance convex hulls generated based on point clouds associated with avoid properties can be used in a collision avoidance optimization algorithm.

[0078] The point clouds associated with context can also be optimized by fitting geometrical features to a group of cloud points. For instance, a group of cloud points of a flat surface may be converted into a plane, a group of cloud points of a three-dimensional objects may be converted into one or more three dimensional figures representing the shape of the point cloud, a group of cloud points representing a circular structure may me converted into a circle, etc.

[0079] Fig. 8 illustrates a virtual environment of the experimental robotic system, where data processing has been performed to some of the cloud points. Similar elements of fig. 4a to 7b have been given the same reference numbers and denoted with a double prime sign " to indicate that the components in figs. 8 are virtual representations of the virtual components of figs. 4a to 7b where data processing have been performed to the cloud points. The cloud points of the rack imitating the door of the CNC machine and associated with avoid properties have been reduced to inliers and converted into a convex hull 325". The cloud points of the box have been associated with avoid properties and reduced to inliers and converted into convex hull 326". The point clouds indicating the pickup area, the placement area and the chuck have been reduced to inliers respectively converted into a rectangular flat surface 323", a rectangular flat surface 328" and a flat circle 327.

[0080] The convex hull 325" of the rack, the convex hull of the box 326", the rectangular flat surface representing the pickup area 323" and the flat circle 327" representing the chuck are provided as parameters for an optimization algorithm configured to search for the optimal position of a robotic arm within the rectangular flat surface represented by the placement area 328". It is noted that the remainder of the cloud points of the robotic system can be disregarded during the optimization process. The optimization process can be carried by any kind of simulation software that can simulate the robotic system and perform optimization based on input parameters.

[0081] In an example, the Advanced Motion Platform provided by the applicant Universal Robots, ref: {2.}, integrated with a Nelder-Mead optimizer according to ref. {3.} is used as simulation software for the optimization process. The result of the optimization is a suggested robot base placement with apercentage of reachable points (taking into consideration collision-free trajectories in between them). In case, the result percentage is below 90%, the system allows the user to change the search space and rerun the optimization.

[0082] The inventors of the present invention have conducted a user study where users with robot deployment experience and simulation experience were asked to perform an optimization process in order to find the optimal position of a robotic arm of the experimental robotic system shown in fig. 3. The users were asked to perform the optimization process by utilizing the robotic system and AR device according to the present invention and to provide context to parts of the robotics system and perform the optimization process as described in in connection with figs. 3-8. Additionally, the users were asked to perform the optimization process utilizing an offline simulation software where the user needed to manually create the virtual environment upon which the optimization algorithm is to be performed. The time that the users spend on performing the optimization process by utilizing the two method / systems were measured. The results of this study were that the users were able to conduct the optimization process utilizing the robotic system and method according to the present invention up to 16 times faster than by utilizing an offline simulation tool. Especially, the robotic system with the AR device made it possible for the user to create the virtual environment upon which the optimization software executes the optimization algorithm much faster than by creating the virtual environment based on an offline simulation software.BRIEF DESCRITPION OF FIGUR REFERENCESREFERENCES{1.} C. B. Barber, D. P. Dobkin, and H. Huhdanpaa, "The quickhull algorithm for convex hulls," ACM Transactions on Mathematical Software, vol. 22, no. 4, pp. 469-483, Dec. 1996. [Online]. Available: https: / / dl.acm.org / doi / 10.1145 / 235815.235821{2.} "Universal Robots Actin UR," [Online], Universal Robots, accessed: https: / / www.universal-robots.com / fi / plus / products / energid / actin-ur / on 2023-09-13.{3.} J. A. Nelder and R. Mead, "A Simplex Method for Function Minimization," The Computer Journal, vol. 7, no. 4, pp. 308-313, 01 1965.

Claims

CLAIMS1. A robotic system comprising:• a sensor system configured to obtain sensor data indicative of one or more properties of the real environment of the robotic system;• an AR. (Augmented Reality) device comprising a display device; the AR device is configured to: o establishing the relative position and orientation of the AR device in relation to a spatial reference system of the robotic system; o establishing a virtual environment based on sensor data obtained by the sensor system; o displaying the virtual environment on the display device based on the relative position and orientation of the AR device in relation to the spatial reference system of the robotic system; o receiving inputs from a user enabling the user to select parts of the virtual environment and associating context to the selected parts;• a control system comprising one or more processing devices, the one or more processing devices being configured to control the robotic system based on the selected parts of the virtual environment and the context associated with the selected parts of the virtual environment.

2. The robotic system according to claim 1 wherein said sensor system comprises a 3D scanner providing sensor data in form of a depth map indicating surface structures of the robotic system.

3. The robotic system according to any one of claims 1-2, wherein said virtual environment is a 3D world space comprising virtual surface elements representing surface structures of the robotic system and said AR device is configured to enable the user to select parts of the virtual surface elements.

4. The robotic system according to any one of claims 1-3, wherein the AR device is configured to select or deselect parts of the virtual environment based on the relative position and orientation of the AR device in relation to the spatialreference system of the robotic system and based on a volume defined in relation to the AR. device.

5. The robotic system according to any one of claims 1-4, wherein said AR device comprises a virtual 3D selecting tool moveable by a user within the virtual environment, where the virtual 3D selecting tool is configured to select or deselect virtual elements within a predefined 3D virtual volume.

6. The robotic system according to any one of claims 1-5 wherein said virtual environment is a point cloud comprising a plurality of cloud points indicating surface structure points of the robotic system in a 3D world space.

7. The robotic system according to claim 6 wherein said AR device is configured to enable a user to select cloud points of said point cloud and to associate context to the selected cloud points.

8. The robotic system according to any one of claims 1-7 wherein the context associated with the selected parts of the virtual environment defines one or more of the following properties:• interact properties indicating structures of the robotic system that the robotic arm and / or an end effector of the robotic arm can interact with;• avoid properties indicating structures of the robotic system that the robotic arm and / or an end effector of the robotic arm must avoid during movement of the robotic arm;• optimize properties indicating structures of the robotic system that can be used as properties that shall be optimized by an optimization algorithm relating to a robot application to be carried out by the robotic system.

9. The robotic system according to anyone of claims 1-8 wherein the control system is configured to execute an optimization process providing at least one optimization parameter relating to the robotic system, wherein said optimization process is based on the selected parts of the virtual environment and the context associated with the selected parts.

10. The robotic system according to any one of claims 1-9, wherein the AR. device is configured to:• display one or more predefined contexts on the display device;• enable a user to select one or more of the predefined contexts; and• enable a user to associate one or more of the selected the predefined contexts to the selected parts of the virtual environment.

11. The robotic system according to any one of claims 1-10 wherein the context associated with the selected parts of the virtual environment defines one or more of the following properties:• state properties defining a state of the selected parts of the virtual environment;• action properties defining action and / or interactions that can be performed by or with the selected parts if the virtual environment;• geometric properties relating to geographical properties of at least a device or element of the robotic system.

12. The robotic system according to any one of claims 1-11 where in the AR device is configured to superpose the virtual environment onto the real environment of the robotic system or a representation of the real environment using the display device.

13. The robotic system according to any one of claims 1-12 wherein the AR. device comprises a camera recording at least a part of the robotic system and the AR device is configured to display the recording on the display device and to superpose the virtual environment onto the displayed recording.

14. The robotic system according to any one of claims 1-13 comprising a robotic arm configured to move in multiple degrees of freedom and at least one additional component.

15. The robotic system according to any one of claims 1-14 wherein the AR device comprises the sensor system.

16. A method of deploying a robotic system, comprising the steps of:• obtaining sensor data indicative of one or more properties of the real environment of the robotic system using at least one sensor system;• establishing a virtual environment based on sensor data obtained by the sensor system;• providing at least one AR (Augmented Reality) device comprising a display device;• establishing the relative position and orientation of the AR device in relation to a spatial reference system of the robotic system;• displaying the virtual environment on the display device based on the relative position and orientation of the AR device in relation to the spatial reference system of the robotic system ;• selecting parts of the virtual environment via the AR device;• associating context to the selected parts of the virtual environment;• deploying at least a part of the robotic system based on the selected parts of the virtual environment and the context associated with the selected parts.

17. The method according to claim 16, wherein the step of obtaining sensor data comprises a step of 3D scanning at least a part of the robotic system and providing sensor data in form of a depth map indicating surface structures of the robotic system.

18. The method according to any one of claims 16-17, wherein said virtual environment is a 3D world space comprising virtual surface elements representing surface structures of the robotic system and where the step of selecting parts of the virtual environment comprises selecting at least a part of the virtual surface elements.

19. The method according to any one of claims 16-18, comprising a step of arranging the AR. device in the real environment and a step of selecting or deselecting parts of the virtual environment based on at least one of the position and the orientation of the AR device in relation to the spatial reference system of the robotic system.

20. The method according to claim 19, comprising a step of selecting or deselecting parts of the virtual environment based on parts of the real environment positioned within a predefined volume defined in relation to the AR device.

21. The method according to any one of claims 16-20 wherein the virtual environment is a point cloud comprising a plurality of cloud points indicating surface structure points of the robotic system in a 3D world space and the step of displaying the virtual environment comprises displaying the point cloud as a 3D world on the display.

22. The method according to claim 21 wherein the step of selecting parts of the virtual environment comprises selecting at least one of the cloud points and the step of associating context to the selected parts of the virtual environment comprises associating context to the selected cloud points.

23. The method according to any one of claims 16-22, wherein the method comprises the steps of:• displaying one or more predefined contexts on the display device;• selecting one or more of the predefined contexts; and• associating one or more of the selected predefined contexts to the selected parts of the virtual environment.

24. The method according to any one of claims 16-23, wherein the step of associating context to the selected parts of the virtual environment comprises associating the selected parts with one or more of the following properties:• interact properties indicating structures of the robotic system that the robotic arm and / or an end effector of the robotic arm can interact with;• avoid properties indicating structures of the robotic system that the robotic arm and / or an end effector of the robotic arm must avoid during movement of the robotic arm;• optimize properties indicating structures of the robotic system that can be used as properties that shall be optimized by an optimization algorithm relating to a robot application to be carried out by the robotic system.

25. The method according to any one of claims 15-24, comprising a step of executing by a processing device an optimization process providing at least one optimization parameter relating to the robotic system, wherein the optimization process is based on the selected parts of the virtual environment and the context associated with the selected parts.

26. The method according to any one of claims 16-25, wherein the step of associating context to the selected parts of the virtual environment comprises associating the selected parts with one or more of the following properties:• state properties defining a state of the selected parts of the virtual environment;• action properties defining action and / or interactions that can be performed by or with the selected parts if the virtual environment;• geometric properties relating to geographical properties of at least a device or element of the robotic system.

27. The method according to any one of claims 16-26, comprising a step of superposing the virtual environment onto the real environment of the robotic system or a representation of the real environment.

28. The method according to any one of claims 16-27, comprising a step of obtaining a recording of at least a part of the robotic system utilizing a camera of the AR device and displaying the recording on a display device of the AR device and superposing the virtual environment onto the displayed recording.

29. The methods according to any one of claim 16-28, comprising a step of providing a robotic arm and at least one additional component.

Images