Method for inspecting and / or handling components via a robotic arm and corresponding system and computer program product - Patents.com
Patent Information
- Application Number
- JP2024542957
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-02-23
- Filing Date
- 2023-02-22
- Publication Date
- 2025-03-26
Smart Images
Figure 2025508311000001_ABST
Abstract
Description
[Technical field]
[0001] Description text Various embodiments of the present disclosure relate to a solution for controlling the movement of a robotic arm. [Background technology]
[0002] FIG. 1 shows an example of a control system for a robotic arm 10, which typically comprises a series of rigid segments or links connected by revolute or translation joints actuated by respective motors.
[0003] In particular, in the example considered, the system needs to perform one or more machining and / or inspection operations of the component 20. For this purpose, a given tool and / or sensor needs to be placed at a given position, mounted on the robot arm 10, typically on the last link of the robot arm 10, typically identified via a so-called tool centre point (TCP).
[0004] Consequently, to control the movement of the robot arm 10, the system includes a robot arm controller 30 configured to drive the motors of the robot arm 10 so as to actuate the joints of the robot arm 10 and to position the links and consequently the TCP of the robot arm 10. To this end, the controller 30, implemented for example via a programmable logic controller (PLC) or some other processing circuit, executes actuation instructions that typically control the drive of the motors of the robot arm 10, typically the acceleration, velocity and position profiles, also in response to one or more feedback signals indicative of the position and / or movement of the robot arm 10, such as signals provided by encoders associated with the joints and / or gyroscopes and / or accelerometers mounted on one or more of the links.
[0005] In many cases, modern controllers 30 are also able to receive control commands at a higher level, for example commands containing the required position and typically also the required orientation for the TCP, or to receive directly the trajectory to be followed. Indeed, by using a kinematic model of the robot arm 10, the controller 30 is able to generate a sequence of actuation instructions to reach the required position and the required orientation. For example, for this purpose, documents US 5,737,500 A and US 6,400,998 B1 can be mentioned, the contents of which are incorporated herein by reference.
[0006] In this context, modern robot arms 10 often have kinematic redundancy, i.e. the size of the working space is smaller than the size of the joint space. In fact, this makes it possible to reach a given position and a given orientation with multiple and practically infinite solutions that can be optimized. However, this also implies additional computational and control complexity. For example, in this case, it is not easy to predict all the movements of the links of the robot arm 10 according to a control program that includes a series of actuation instructions and / or control commands sent to the controller 30. For example, the most common programming languages for the movements of robot arms are PDL2 (Comau), RAPID (ABB), KRL (Kuka) or AS (INFORM and Kawasaki).
[0007] Consequently, a solution has been proposed that allows a simulation of the movements of the robot arm 30 in order to check the behavior of the robot arm 10 according to a control program transmitted to the controller 30. For example, for this purpose, a computing device 40, such as a computer, is typically used, which allows a simulation of the movements of the robot arm 10 to be displayed in a virtual environment 42. For example, for this purpose, reference may be made to the document US 2008 / 0125893 A1, the contents of which are incorporated herein by reference.
[0008] For example, FIG. 2 illustrates typical operation of a program executing on a computer 40.
[0009] In particular, after a start stage 1000, the computer 40 receives in stage 1002 a model RM of the robot arm 10 and a model CM of the component 20. For example, the model RM may include a three-dimensional (3D) model of the robot arm 10 and a respective kinematic model. Similarly, the model CM may include a 3D model of the component 20 and possibly further features of the component 20, such as properties of the material of the component 20 that may be important for possible machining operations to be performed, such as properties related to deformability. For example, the 3D model may be generated using a conventional CAD (Computer-Aided Design) program. Consequently, the model RM of the robot arm and the model CM of the component can be loaded into the virtual environment 42 together with their corresponding positions; i.e. the models RM and CM are placed in the virtual environment 42 based on the relative positions between the robot arm 10 and the component 20 in the real environment. In general, the virtual environment 42 may also include further elements, such as walls and / or fences that limit the movement of the robot arm 10, as well as elements related to possible obstacles in the environment in which the robot arm 10 is placed.
[0010] In step 1006, the operator can then generate the control program RPRG, which comprises a series of actuation instructions and / or control commands. Generally, the simulation program itself or a further development environment can be used for this purpose. Consequently, in step 1008, the computer 40 receives the control program RPRG and simulates the movements of the robot arm 10 in the virtual environment 42 by using the models RM and CM for this purpose.
[0011] Consequently, in step 1010, the operator or directly the computer 40 can check whether the control program RPRG is correct. For example, for this purpose, the computer 40 can check whether the movements of the model RM of the robot arm involve collisions with objects in the virtual environment 42, such as components CM, walls, etc.
[0012] If the control program RPRG is correct (output "yes" from verification step 1010), the operator, or directly the computer 40, can then send the control program RPRG to the controller 30 in step 1012 and the method ends in final step 1014. If not (output "no" from verification step 1010), the operator can modify the control program RPRG in step 1006 and repeat the simulation in step 1100.
[0013] Consequently, in the example considered, steps 1002 to 1010 implement an offline (i.e., without any real interaction with the robot arm 10) programming method 1100, while only step 1012 implements an online control procedure 1200, which results in the execution of the control program RPRG by the controller 30 and a corresponding movement of the robot arm 10.
[0014] Consequently, programming via the virtual environment 42 makes it possible to reduce the risk of problems arising during the actual movement of the robot arm 10. However, such programming is carried out using complex programming languages and therefore requires a suitable operator. Summary of the Invention
[0015] Therefore, an object of various embodiments of the present disclosure is a new solution that facilitates the programming of operations performed by a robotic arm.
[0016] According to one or more embodiments, one or more of the above objects are achieved through a method having the characteristic elements specifically recited in the following claims. The embodiments further relate to a corresponding system and to a corresponding computer program product, which can be loaded into the memory of at least one computer and comprises portions of software code for implementing the steps of the method when the product is executed on the computer. As used herein, reference to such a computer program product is intended to be equivalent to reference to a computer-readable means comprising instructions for controlling a processing system to coordinate the execution of the method. The reference to "at least one computer" is obviously intended to highlight the possibility that the present disclosure may be implemented in a distributed / modular manner.
[0017] The claims form an integral part of the technical teachings of the description provided herein.
[0018] As previously mentioned, various embodiments of the present description relate to a solution for inspecting and / or handling components or objects via a robotic arm, where the movements of the robotic arm are managed via a controller in response to motion instructions. For example, the motion instructions can specify one or more points of a trajectory in the reference system of the robotic arm. In various embodiments, one or more sensors are mounted on the robotic arm and / or installed on a platform on which the robotic arm is mounted and / or installed in the environment in which the robotic arm is located. The controller and the one or more sensors are connected to a computer.
[0019] In particular, in various embodiments, the computer receives a three-dimensional model of the component and depicts the three-dimensional model of the component in a virtual environment having a given reference system. As described in more detail below, in various embodiments, the three-dimensional model of the component can be generated and / or updated by acquiring one or more images and / or point clouds of the component via a sensor. In various embodiments, the virtual environment allows for designation of one or more points of interest. As a result, the computer receives the one or more points of interest in the reference system of the virtual environment. In various embodiments, the computer can also automatically generate one or more of the points of interest depending on, for example, a given area or surface of the component selected by an operator.
[0020] The computer then presents a graphic interface that allows the user to specify a series of commands, including a plurality of commands for interacting with the robot arm and / or one or more sensors, where each command includes data identifying an action and one or more parameters for the action. In various embodiments, the series of commands includes data specifying, for each command, a respective condition indicating when the respective action needs to be performed. For example, the graphic interface may include a graphic interface for specifying a list of commands and / or a graphic interface for specifying the series of commands via a flow chart. In particular, in various embodiments, the first command requests a movement of the robot arm to a given one of the one or more points of interest. For example, the series of commands may be saved in the form of a list, for example by means of an Excel, CSV, XML file. In various embodiments, the computer may also automatically generate one or more of the commands, for example depending on a given type of inspection performed for all points of interest or a subset of points of interest.
[0021] In various embodiments, the computer then acquires one or more images and / or point clouds of the component via one or more sensors and compares the images and / or point clouds with the three-dimensional model of the component to determine the position of the component relative to the robot arm. The computer can then convert coordinates of a given interest point in the reference system of the virtual environment to coordinates in the reference system of the robot arm, using the position of the component relative to the robot arm for this purpose. In various embodiments, the computer can also update the three-dimensional model of the component depending on the images and / or point clouds. In particular, in various embodiments, the computer can perform a first scan of the component, and possibly of the environment, to generate a model of the component and / or to update the model of the component. The operator can then specify the interest points and commands using the model of the component above. Consequently, in this case, the computer can perform a second scan of the component to align the model (with the respective interest points) with the (real) position of the component.
[0022] In various embodiments, the computer then uses the 3D model of the component and the model of the robot arm to generate a second virtual environment. In general, this virtual environment may correspond to the virtual environment used to define the points of interest (manually or automatically), or a dedicated virtual environment may be used.
[0023] In various embodiments, the computer can also acquire, via one or more sensors, one or more images and / or a point cloud of one or more (further) obstacles in the environment in which the robot arm is placed. Consequently, the computer can also generate a three-dimensional model of the one or more obstacles according to the one or more images and / or the point cloud, and place the three-dimensional model of the one or more obstacles in the second virtual environment.
[0024] In various embodiments, the computer then iterates through a sequence of operations for each command. In particular, first the computer checks whether the command requires a movement of the robot arm, since, for example, the command corresponds to a first command that requires a movement of the robot arm to a given point of interest. Other movement commands may require a movement to a point determined according to a previous command, a pre-determined movement, and / or opening a file containing a sequence of one or more movement instructions and sending one or more movement instructions to the controller. The one or more commands may also require acquisition of data via one or more sensors and / or activation of one or more actuators mounted on the robot arm and configured to manipulate the components. For example, the commands may require at least one of: taking a sound recording and storing the sound recording in a file; taking a sound recording, generating a text via speech recognition of the sound recording, and comparing the recognized text to a reference text; taking an image and storing the image in a file; and taking an image, generating a text via a character recognition operation on the image, and comparing the recognized text to a reference text. Additionally or alternatively, the commands may call for the actuation of one or more of the actuators, and / or the simultaneous actuation of one or more of the actuators and the acquisition of data via one or more sensors.
[0025] In case the command corresponds to the first command (or similarly another command to reach a given position), the computer automatically selects a trajectory for moving the robot arm in the coordinates of the given interest point in the reference system of the robot arm and evaluates the selected trajectory in the second virtual environment to determine whether the robot arm can follow the selected trajectory without colliding with the three-dimensional model of the component and possibly with three-dimensional models of one or more (further) obstacles. For example, for this purpose, the computer can define an optimization problem by adding as constraints all obstacles present in the second virtual environment and constraints resulting from the process (e.g. the position and possibly orientation of the interest point to be reached, the distance from the surface of the component, etc.), and can optionally find an optimized trajectory according to a given cost function such as the distance covered, the inertia of the joints, the energy consumption or a combination thereof. Consequently, in various embodiments, the computer can select different trajectories and evaluate / simulate whether these trajectories satisfy the basic constraints (non-collision trajectories to reach one or more required points). The computer can then calculate the cost function and select the trajectory with the lowest cost.
[0026] Consequently, once the computer selects a given trajectory, particularly if the robot arm can follow the selected trajectory without colliding with components and obstacles, the computer automatically generates one or more movement commands for the selected trajectory and transmits the one or more movement commands to the controller.
[0027] Consequently, in this manner, the operator simply specifies points of interest and corresponding movements on the three-dimensional model of the component, and the computer automatically determines the alignment of the component relative to the robot arm, determines the trajectory to follow to execute a given command, and generates the respective instructions that the controller will then execute. [Brief description of the drawings]
[0028] Embodiments of the present disclosure will now be described with reference to the accompanying drawings, given purely by way of non-limiting example. [Figure 1] 1 shows an example of a control system for a robot arm. [Diagram 2] 2 shows an example of the operation of the computer of the control system of FIG. 1 . [Diagram 3] 1 illustrates an embodiment of a control system for a robotic arm according to the present disclosure. [Figure 4] 4 illustrates an embodiment of a sensor and / or manipulator that can be mounted on the robot arm of FIG. 3. [Diagram 5] 4 shows a flow chart of an embodiment of the operation of the computer of the control system of FIG. 3. [Figure 6] 6 illustrates an embodiment of an offline phase of the operational method of FIG. 5. [Figure 7] The operational steps of FIG. 6 are shown in detail. [Figure 8] The operational steps of FIG. 6 are shown in detail. [Figure 9] The operational steps of FIG. 6 are shown in detail. [Figure 10] The operational steps of FIG. 6 are shown in detail. [Figure 11] The operational steps of FIG. 6 are shown in detail. [Figure 12] 6 illustrates an embodiment of the online phase of the operational method of FIG. 5. [Figure 13A] 13 shows details of the operational steps of FIG. 12. [Figure 13B] 13 shows details of the operational steps of FIG. 12. [Figure 13C] 13 shows details of the operational steps of FIG. 12. [Figure 13D] 13 shows details of the operational steps of FIG. 12. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0029] In the following description, numerous specific details are provided to enable a thorough understanding of the embodiments. The embodiments may be implemented without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known operations, materials, or structures are not shown or described in detail to avoid obscuring aspects of the embodiments.
[0030] Throughout this description, references to "an embodiment" or "one embodiment" mean that a particular feature, characteristic, or structure described with reference to the embodiment is included in at least one embodiment. Thus, the appearances of phrases such as "in an embodiment" or "in one embodiment" at various points in this description do not necessarily all refer to one and the same embodiment. Also, particular features, characteristics, or structures may be combined in any suitable manner in one or more embodiments.
[0031] References used herein are provided merely for convenience and, as a result, do not define the scope of protection or the scope of the embodiments.
[0032] In the following Figures 3 to 13, parts, elements or components already described with reference to Figures 1 and 2 are designated by the same references previously used in the preceding Figures; in order not to overload the detailed description of this specification, the description of the above elements previously described will not be repeated below.
[0033] As previously mentioned, the present description provides a simpler solution for controlling the movement of a robotic arm.
[0034] FIG. 3 shows an embodiment of a control system for a robotic arm 10 according to the present description.
[0035] In the embodiment considered, the movements of the robot arm 10 are controlled via a controller 30. For a general description of the operation of the robot arm 10 and the controller 30, reference may be made to the previous description of FIG.
[0036] In the embodiment considered, the robot arm 10 is equipped with sensors 102 that can be used for inspection of the component 20 and / or actuators 104 that can be used for machining or generally for handling the component 20 .
[0037] 4, the sensor 102 may include one or more cameras 102a configured to acquire images of at least a portion of the component 20, possibly even in different spectra, e.g., in the visible and / or infrared spectrum. In various embodiments, the cameras 102a may also be used to generate a depth image (or point cloud) of the component 20. In general, other sensors, such as LIDAR (light detection and ranging) sensors and / or radar systems, configured to provide data indicative of the shape of the component 20 may also be used in addition to or instead of the camera 102a.
[0038] In various embodiments, the sensor 102 may also include other sensors capable of detecting physical characteristics of the component 20. For example, with respect to inspection of metal sheets, the sensor 102 may include at least one of an ultrasonic transducer that can be used, for example, to check for defects in the material and / or variations in the thickness of the metal sheet; and / or a test hammer and an acoustic sensor, such as a microphone 102b, that can be used for so-called tap testing.
[0039] Alternatively, the actuator 104 may include any actuator capable of manipulating, e.g., performing machining, and / or disassembling and / or assembling the component 20. For example, a gripper 104a and an electric wrench 104b are illustrated diagrammatically in Fig. 4. However, those skilled in the art will appreciate that the actuator 104 may also include other types of tools, such as a welder (a spot welder, a laser welder, or a welding gun), other gripping members (e.g., suction means), tools for applying a substance (e.g., an adhesive or paint) onto the component 20, etc.
[0040] In general, the sensors 102 (possibly in cooperation with one or more of the actuators 104) may not only determine or verify physical properties of the mechanical component 20, but may also detect electrical properties or even more complex responses of the component 20. For example, for testing an electrical and / or electronic component 20, the sensors 104 may include sensors for measuring electrical properties of the component, e.g., voltage, current, and / or resistance. Alternatively, for more complex electronic components / devices 20, such as a mobile phone, notebook, or vehicle infotainment system, the sensors 102 and actuators 104 may also be used to interact with the electronic device 20, e.g., pressing a button on the electronic device, playing an audio file via the speaker 104c, monitoring a visual response of the electronic device via the camera 102a or an acoustic response via the microphone 102b, etc.
[0041] Generally, one or more sensors may also be installed in the environment of the component 20. For example, FIG. 3 is a schematic illustration of a camera and / or LIDAR sensor 12. Similarly, the environment may include actuators, e.g., a conveyor belt for transporting the component 20 to the robot arm 10. However, in various embodiments, the robot arm 10 may be mounted on a mobile platform 14 that allows the robot arm 10 to be moved in the environment, e.g., to move the robot arm 10 next to the component 20 and / or to move the robot arm 10 relative to the component 20. Consequently, in various embodiments, one or more of the sensors 104 used for inspection of the component 20 may be installed in the environment and / or mounted on the mobile platform 14. For example, in various embodiments, the platform 14 may be implemented in the form of an Automated-Guided Vehicle (AGV) or an Autonomous Mobile Robot (AMR). For example, the use of a mobile platform 14 is particularly advantageous when the component 20 being inspected or handled forms part of a larger object, such as a motor vehicle.
[0042] In the embodiment considered, various sensors (e.g., 102 and 12) and actuators (e.g., 104 and 14) are operatively connected to the computer 40a. In general, any communication system can be used for this purpose, including wired connections, e.g., via CAN bus and / or Ethernet, and / or wireless connections, e.g., via Wi-Fi and / or mobile networks. In general, the computer 40a can communicate with the controller 30 via this communication system or also via additional communication channels. Also, in various embodiments, the computer 40a can communicate with at least some of the sensors 102 and / or actuators 104 via the controller 30. Finally, the operations of the computer 40a can also be implemented via a processing system including one or more computers, which can be local (i.e., located in the environment in which the robot arm is installed or installed on the platform 14) and / or remote. Similarly, at least some of the operations of the controller 30 can also be implemented in the computer 40a.
[0043] As illustrated in FIG. 3, in various embodiments, the computer 40a implements a development environment 42a that allows for controlling the operation of the controller 30 and thus the robotic arm 10 through appropriate programming, for example using software code.
[0044] 5 shows an embodiment of the development environment 42a. In particular, after a start phase 2000, the development environment 42a allows to specify the operation of the robot arm 10, the sensors 12 / 102 and the actuators 14 / 104 in an offline phase 2100, i.e. without any interaction with the controller 30 and / or the robot arm 10. The environment 42a then assumes an online phase 2200, during which the computer 40a takes charge of the robot arm 10, drives one or more of the actuators 14 / 104 and receives data from one or more of the sensors 12 / 102. In general, the operation can then be finished or it can then be repeated for another component 20 (as illustrated diagrammatically in FIG. 5).
[0045] FIG. 6 illustrates a first embodiment of the online phase 2100.
[0046] When step 2100 begins, the computer 40a receives, in step 2102, a model CM of the component 20, including, among other things, a 3D model of the component 20, such as a CAD model. As also illustrated in FIG. 7, the computer 40a, in step 2102, shows the 3D model of the component 20 in a virtual environment 42b. For example, similar to a conventional CAD environment, the virtual environment 42b may allow the 3D model to be displayed, rotated and / or zoomed in on a display, etc. As described in more detail below, in various embodiments, the three-dimensional model CM of the component 20 may be generated and / or updated by acquiring one or more images and / or point clouds of the component 20 via a sensor.
[0047] In step 2104, the operator can then specify one or more points of interest CPOI in the virtual environment 42b; i.e. the computer 40a is configured to receive one or more points of interest CPOI in step 2104. In particular, for this purpose the operator selects a given point in the 3D model, for example identified via Cartesian coordinates x, y and z of the virtual environment 42b relative to a given reference point REF. For example, as illustrated in FIG. 7, the reference point REF may correspond to a given position relative to the reference system of the CAD, or in general, to the model CM of the component 20. In particular, typically the points CPOI are located on the surface of the 3D model of the component 20 and / or at a given position relative to the surface of the model CM of the component 20. In various embodiments, the computer 40a also makes it possible to specify, for each point of interest CPOI, respective information of an orientation, for example orientation data including the so-called robot Euler angles, or pitch, roll and yaw.
[0048] In various embodiments, the computer 40a may also enable selection of an area, e.g., a given surface of the model CM of the component 20, and the computer 40a may automatically determine multiple points of interest CPOI depending on the selected area, e.g., placing multiple points of interest CPOI at (approximately) equal distances in the selected area.
[0049] Consequently, in various embodiments, the computer 40a receives in step 2104 a list including at least one point of interest CPOI, where each point of interest CPOI is identified via respective coordinates, e.g., x, y, and z relative to a reference point REF, and possibly orientation data. For example, in FIG. 7, the operator selects four points CPOI1, CPOI2, CPOI3, and CPOI4. In various embodiments, the list may also include a field that allows specifying a unique identifier for each point of interest. For example, for simplicity, it is assumed that points CPOI1, CPOI2, CPOI3, and CPOI4 have identifiers "CPOI1", "CPOI2", "CPOI3", and "CPOI4", respectively.
[0050] In step 2106, the operator then opens a programming environment 42c (see also FIG. 8) in which the operator enters and / or modifies a program CPRG consisting of a sequence of commands CMD. In general, the programming environment 42c can be integrated with the virtual environment 42b in a single program / development environment 42a, or two separate programs / environments can be envisaged.
[0051] In particular, in various embodiments, the development environment allows for the specification of high level commands CMD. For example, Figure 9 shows an example of a program CPRG. In particular, in various embodiments, each command CMD: A unique identifier SID that specifies the number of each command CMD; ·Conditions CON that specify when the respective command CMD is started; a type of action AT that specifies the action to be performed for each command CMD; and one or more fields ARG allowing the specification of one or more parameters of each action AT, e.g. fields ARG1 and ARG2 Includes.
[0052] In general, the unique identifier SID can also be specified implicitly via the number of the command CMD, for example the line number.
[0053] In particular, in various embodiments, at least one kind of action AT comprises a value specifying a movement of the robot arm 10, where the parameter ARG allows to indicate one of the points of interest CPOI. For example, in various embodiments, a generic command with the action AT=“MOVE” is used, where a first parameter ARG1 indicates the fact that the movement has to reach the point of interest CPOI, for example by specifying the value “POI” as parameter ARG1, and a second parameter allows to specify an identifier of one of the points CPOI. However, in general, an action AT, for example “MOVEPOI”, may specifically relate to a movement at a point of interest. Consequently, in this case, the first parameter ARG1 may directly specify an identifier of one of the points CPOI.
[0054] In various embodiments, the computer 40a may also (at least partially) automatically generate the sequence of commands CPRG. For example, as mentioned before, the computer 40a may also enable the selection of an area, such as a given surface of the model CM of the component 20, and the computer 40a may automatically determine a number of points of interest CPOI depending on the selected area. As a result, in this case, the computer may generate a sequence of commands CPRG to reach all points of interest CPOI determined for the selected area in sequence. Also, by indicating a given inspection or a given handling operation for the area, the computer may also automatically generate the respective inspection and / or handling commands to be performed at the points of interest CPOI.
[0055] In general, the list of possible other actions AT depends on the sensors 102 and actuators 104 mounted on the robotic arm 10. Examples of possible actions for testing an electronic device 20 that includes a touch screen, such as a mobile phone, a tablet, or a vehicle infotainment system, are described below.
[0056] For example, in this case the sensors 102 may include a camera 102a and a microphone 102b, and the actuators may include a speaker 104c and an actuator for operating a touch screen, for example in the form of a pen or a finger.
[0057] For example, in this case the action of the movement "MOVE" may include a second sub-action, for example specified via argument ARG1="Touch", where the robot arm 10 needs to touch the touch screen with a pen or a finger, for example by specifying the location to be touched in parameter ARG2.
[0058] In various embodiments, the action "MOVE" may also make it possible to specify movement commands as for programming of a conventional robot arm. However, since the command CMD is simple, in this case the first argument ARG1 specifies only the fact that it is a movement program, for example by using ARG1="Robot Program", and the parameter ARG2 specifies the name of a file containing the respective instructions, for example using the PDL2 language. In particular, in various embodiments, the program instructions refer to a position REF of the 3D model of the component 20 and / or relate to movement instructions relative to the current position of the robot arm 10. For example, as known per se, the movement commands can be entered manually and / or automatically recorded by moving the robot arm 10 manually or via a user interface, i.e. a so-called teach pendant.
[0059] Consequently, in the embodiment considered, the action "MOVE" includes actions that suggest a movement of the robot arm 10, Movement to a given point of interest CPOI; Predetermined movements such as the subaction "touch"; and / or For example, using the sub-action "Robot Program" to control the movement of a conventional robot arm 10 may include.
[0060] Similarly, other actions AT may refer to the use of the camera 102a, microphone 102b, and / or speaker 104c. For example, in various embodiments, the action "Look" controls the operation of the camera 102a and includes the following sub-actions, specified via the parameter ARG1: a subaction "Get" in which the computer 40a retrieves an image from the camera 102a, where the parameter ARG2 may specify the index or name of one of the cameras 102a (if multiple cameras 102a are provided); a subaction "Image" in which the computer 40a retrieves one or more images and locates the image(s) identified, for example, via a parameter ARG2 specifying the file name of the image; and / or a subaction "Text" in which the computer 40a acquires one or more images and searches for the location of a given text, identified via the parameter ARG2, using for this purpose an OCR (Optical Character Recognition) algorithm; may include one or more of:
[0061] Similarly, in various embodiments, the action "Listen" controls the operation of the microphone 102b and includes the following sub-actions specified via the parameter ARG1: a subaction "Sound" in which the computer 40a gets the recording and checks whether it contains an acoustic profile identified, for example via a parameter ARG2 specifying the filename of the sound file; and / or a subaction "Voice" in which the computer 40a gets the recording and checks whether it contains a given text, identified via the parameter ARG2, using for this purpose a speech recognition algorithm; may include one or more of:
[0062] Finally, in various embodiments, the action "Reproduce" controls the operation of the speaker 104c and includes the following sub-actions, specified via the parameter ARG1: a subaction "Sound" in which the computer 40a plays a sound file identified via the parameter ARG2; and / or a subaction "Voice" in which the computer 40a plays back a given text identified via the parameter ARG2, using for this purpose a speech synthesis method; may include one or more of:
[0063] As a result, the instructions shown above make it possible to test the most common functions of the electronic device 20, including the touch screen. For example, Figure 9 shows an example of a program CPRG, where: - The command CMD1 requests speech synthesis of the text "Call Mark"; The command CMD2 requests to compare the following audio profile with the audio profile specified in the audio file callOK.mp3, e.g. the profile corresponding to the sound of a telephone call; The command CMD3 corresponds to an optional waiting phase; · Command CMD4 requests movement to position CPOI1; · The command CMD5 requests to check whether the images acquired for each position contain the image "close.jpg"; The command CMD6 specifically requests a touch on the screen at the location of the image "close.jpg" found with the command CMD5 (ARG2=5); The command CMD7 requests checking whether the image acquired for each position contains the text "Call Terminated"; and Command CMD8 jumps back to command CMD4 if the result of command CMD7 is not positive.
[0064] Consequently, as illustrated in the example, the condition CON allows to specify the condition when the respective command CMD is executed and may include the identifier of the command CMD and the desired result, e.g. "OK" or "NOK". To allow more complex sequences, in various embodiments, the action AT, e.g. "Go To", also allows to specify the identifier of the command CMD to which the program CPRG should jump directly. Consequently, in this way, a subprogram may be implemented conditional on one or more results of a previous command CMD.
[0065] Alternatively, for testing metal sheets, the camera 102a can again be controlled using the action "Look", for example by the camera 102a, the tap test hammer, and / or the ultrasonic transducer. Alternatively, in various embodiments, the action "Tap" can be controlled using the following sub-actions, specified, for example, via the parameter ARG1: a subaction "Sound" in which the computer 40a operates the tap test hammer, obtains a recording and checks whether the recording contains an acoustic profile identified, for example, via the parameter ARG2, which specifies the filename of the sound file; and / or A subaction "Get" in which computer 40a operates the tap test hammer, obtains a recording, and saves the recording with a file name indicated via parameter ARG2. The actions of tapping the test hammer and acquiring the respective audio files can be controlled in parallel by one or more of the following:
[0066] Similarly, in various embodiments, the action "Ultrasound" may include the following sub-actions, specified, for example, via the parameter ARG1: a sub-action "Get Thickness" in which the computer 40a measures, via a transducer, the thickness of the metal sheet and stores the result in a file indicated via the parameter ARG2; and / or A sub-action "Thickness" in which the computer 40a measures the thickness of the metal sheet via a transducer and checks whether the metal sheet has a given thickness. may control the operation of the ultrasonic transmitter and the acquisition of corresponding signals by the ultrasonic receiver in parallel.
[0067] As a result, the previously mentioned instructions make it possible to test the mechanical characteristics of a metal sheet, for example a metal sheet of a vehicle or an aircraft. For example, FIG. 10 shows an example of a program CPRG, in which: · Command CMD1 requests movement to position CPOI1; · The command CMD2 requests to get the image saved as "cpoi1.jpg"; · The command CMD3 requests a tap test to check whether each audio profile corresponds to the file "cpoi1.mp3"; · The command CMD4 requests the operation of the ultrasonic transducer to check if the thickness of the metal sheet corresponds to 200 μm (ARG2=200); · Command CMD5 requests movement to position CPOI2; · The command CMD6 requests to get an image which is saved as "cpoi2.jpg"; The command CMD6 requests a tap test and checks whether each audio profile corresponds to the file "cpoi2.mp3"; and The command CMD8 requests the operation of the ultrasonic transducer to check if the thickness of the metal sheet corresponds to 200 μm (ARG2=200).
[0068] Consequently, in the embodiment considered, the computer 40a can save the program CPRG in a step 2108. In particular, in various embodiments, the program CPRG corresponds to a list of commands, where each command CMD includes a given number of fields. Said program CPRG corresponds to a high-level programming metalanguage and can be saved in any list or table format, for example in the form of an Excel file, a CSV file, an XML file, etc. Finally, the step 2100 ends in a final step 2110. Similarly, the computer 40a stores the data of the points of interest CPOI in a step 2108. For this purpose, any list or table format can also be used, such as an Excel file, a CSV file, an XML file, etc.
[0069] In addition to or instead of the specification of the system's behavior via a list of commands CMD, as illustrated in Fig. 11, the development environment 42c may include a graphic interface 42d for specifying the operation of the program CPRG via a flowchart. For example, for this purpose an operator may draw a conventional flowchart including stages / processes and verification / decision steps.
[0070] In particular, in various embodiments, the interface 42c allows for specifying for each step / process a respective action AT and a respective parameter ARG, for example via a window showing the properties of the step / process. For example, the steps / processes for the commands CMD1 to CMD7 of FIG. 9 may be provided in this way.
[0071] In particular, in the embodiment considered, the interface does not allow specifying the conditions CON for each step / process, but a verification / decision step is provided which makes it possible to verify the result of the previous step / process. For example, a verification / decision step can thus be envisaged for the command CMD8.
[0072] Consequently, in various embodiments, the computer receives data identifying a flow chart in step 2106 and converts the flow chart into a corresponding sequence of commands CMD, thus generating the program CPRG. Similarly, the computer 40a may receive a program CPRG and automatically generate a corresponding flow chart to facilitate understanding of the program CPRG. Consequently, in various embodiments, the development environment may be capable of switching between a view 42c in the form of a list and a view 42d in the form of a flow chart for the same program CPRG.
[0073] Consequently, in comparison to the operations of FIG. 2, in various embodiments, the development environment 42a does not simulate the operation of the robot arm 10 during stage 2100, but rather the operator can specify the movement of the robot arm 10, the acquisition of data via the sensors 102, and / or the actuation of the actuators 104 via simple high-level commands and / or flow charts.
[0074] FIG. 12 shows an embodiment of the online phase 2200, ie the control of the robot arm 10 and the sensors 102 and / or actuators 104.
[0075] In particular, when stage 2200 is started, the computer 40a acquires, in stage 2202, a model CM of the component 20 and further data acquired by the sensors 102 mounted on the robot arm 10 and / or by the sensors 12 installed in the environment and / or by sensors mounted on the platform 14. In particular, in stage 2202, the computer 40a is configured to reconstruct a 3D model of the component 20, using for this purpose the data provided by the above sensors. Similarly, the computer 40a can reconstruct a 3D model of the robot arm 10 and of the environment in which the component 20 is located, for example with reference to possible obstacles. Consequently, for this purpose the computer 40a may use one or more cameras, LIDAR sensors and / or radars mounted on the robot arm 10 and / or in the environment and / or on the platform 14.
[0076] In various embodiments, the computer 40a is configured to compare the model CM with the reconstructed model in step 2202 to identify a reference position REF of the model CM of the component 20 in the reference system of the robot arm 10. For example, for this purpose, the computer 40a can perform a matching operation between the model CM and the reconstructed model. For example, for this purpose, the document "A point sampling algorithm for 3D matching of irregular geometries" by Tolga Birdal and Slobodan Ilic, IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2017 can be cited. As a result, once the position of the robot arm 10 is known and / or the position of the mobile platform 14 is detected, the computer 40a can determine the relative position of the component 20 with respect to the robot arm 10. As a result, once the reference position REF of the model CM and the current position of the component 20 are known, the computer 40a can calculate the position of the reference REF in the reference system of the robot arm 10.
[0077] Optionally, the computer may also generate, in step 2204, a modified / updated model CM' of the component 20 depending on the acquired data and possibly the original model CM.
[0078] For example, this is illustrated in Figures 13A, 13B, 13C and 13D. In particular, Figure 13A shows an example of a model CM of the component 20, Figure 13B shows an example of an image IMG acquired by the camera 102a and / or 12 component CM, and Figure 13C shows an example of a point cloud of the component 20, e.g. a depth image acquired via the stereo camera 12 and / or by moving the camera 102a to different positions and / or using a LIDAR sensor or radar.
[0079] Consequently, in various embodiments, the computer 40a may combine the information IMG and / or PC with the model CM to generate a modified model CM'. For example, in this way it is possible to detect objects that are fixed to the component 20 but that were not envisaged in the original model 20. For example, with respect to a vehicle, the computer 20 may detect other objects fixed to the vehicle, such as a roof rack. Similarly, the computer 40a may then generate and / or update a 3D model of the environment, in particular of possible obstacles located near the robot arm 10 and / or the component 20.
[0080] In particular, in various embodiments, the computer 40a can perform a first scan of the component 20 to generate a model CM of the component 20 and / or obtain an updated model CM' of the component. An operator can then specify points of interest and commands using the model CM or the updated model CM' of the component 20. Consequently, in this case, the computer 40a can perform a second scan of the component to align the model CM (and the respective points of interest CPOI) with the (real) position of the component 20.
[0081] Consequently, at the end of stage 2202, the computer 40a has determined the reference position REF in the reference system of the robot arm 10 and possibly updated the model CM' of the component 20. In stage 2026, the computer 40a then receives the point of interest CPOI, for example by opening the respective file. Consequently, once the reference position REF relative to the reference system of the robot arm 10 and the coordinates of the point CPOI relative to the reference position REF are known, the computer 40a calculates in stage 2206 the coordinates and possibly the orientation information of the point CPOI in the reference system of the robot arm 10.
[0082] In the embodiment considered, the computer 40a also receives the program CPRG in stage 2208, for example by opening the respective file and selecting the first command CMD1. In particular, as mentioned before, the program CPRG typically contains only high-level instructions that do not correspond to instructions for movements of the robot arm 10; i.e. the commands CMD cannot be interpreted by the controller 30. Consequently, in various embodiments, the computer 40a pre-processes the selected command CMD, for example CMD1, in stage 2210 to check whether the respective command CMD requests a movement of the robot arm 10.
[0083] If the selected command does not require actuation of the robotic arm (outputting "yes" at verification step 2212), the computer 40a proceeds to step 2214, where it interacts with the sensors 102 and / or actuators to, for example, perform commands to acquire data from one or more sensors 102 (and possibly process this data, e.g., process acquired images or acquired audio files) and / or to actuate one or more actuators 104 (e.g., play an audio file, perform a tap test, and / or detect thickness).
[0084] Alternatively, in the case where the selected command calls for actuation of the robot arm 10, the computer 40a automatically determines the instructions RPRG of the controller 30, i.e., instructions in the programming language of the controller 30, to obtain the requested movement, in step 2210. For example, in various embodiments, the computer 40a generates movement instructions that specify respective target / set points, e.g., via Cartesian coordinate data or data in joint space, and the controller 30 is configured to solve the inverse kinematics.
[0085] Consequently, the instruction sequence RPRG corresponds to a subprogram which contains only the instructions for the movements of the robot arm 10 required for the selected command.
[0086] In particular, for this purpose, the computer 40a generates in a stage 2210 a virtual environment and places the model RM of the robot arm 10 in the virtual environment, possibly also including 3D models of other obstacles in the environment. In general, this virtual environment may also correspond to the virtual environment 42a used to determine the interest point. The computer 40a also places the model CM (preferably the updated model CM') in the virtual environment, using for this purpose the position of the reference REF determined in stage 2202, i.e. the position of the model CM relative to the reference system of the robot arm 10. Consequently, in various embodiments, the computer 40a automatically places the models RM and CM (and possible models of further obstacles) based on the relative positions detected in stage 2202.
[0087] Consequently, once the position of the robot arm 10 is known (possibly by obtaining corresponding data from the controller 30 and / or the mobile platform 14), the computer 40a can, in stage 2210, evaluate the movement of the model RM of the robot arm 10 for one or more trajectories to perform the requested movement action, for example to perform the action "Touch" or to reach the point of interest CPOI to perform a predetermined program using the argument "Robot Program".
[0088] For example, in various embodiments, the program RPRG may correspond to instructing the controller 30 to follow a given trajectory, for example identified via a series of set points. Consequently, in this case, the computer 40a can select the trajectory, generate the respective movement instructions RPRG, and evaluate / simulate the movement, in step 2210.
[0089] Consequently, in this case, the computer 40a can verify in step 2212 whether the selected trajectory implements the requested movement, for example because the (simulated) robot arm executes the requested movement and does not collide with objects in the virtual environment, for example the model CM (or preferably CM') of the component 20 and / or with models of one or more other objects in the environment. Consequently, if the selected trajectory fails to implement the requested movement (outputting "no" from the verification step 2212), the computer returns to step 2210 to select another trajectory and / or generate another instruction sequence RPRG. Consequently, the computer 40a generates, via steps 2210 and 2212, a trajectory that avoids the obstacle (collision avoidance), using for this purpose a scene including the models CM and RM positioned via aligning the models CM to information acquired via the sensors 102 / 12, and possibly one or more models of other objects in the environment.
[0090] For example, for this purpose, the computer 40a may define an optimization problem in stage 2212 by adding as constraints all obstacles present in the second virtual environment and constraints resulting from the process (e.g., the position of the interest point or points of interest to be reached, and possibly possible constraints on the orientation, the surface of the component and / or the distance from other obstacles, etc.), and may optionally select an optimized trajectory according to a given cost function such as the distance covered, the inertia on the joints, the energy consumption, or a combination thereof. As a result, in various embodiments, the computer 40a may determine in stage 2212 one or more trajectories (collision-avoiding trajectories that reach one or more required points) that satisfy the basic constraints. Then, in various embodiments, the computer may calculate the cost function and select the trajectory with the lowest cost. Consequently, in cases where the computer 40a is not able to determine a trajectory capable of implementing the requested movement (outputting "No" from verification step 2212), for example because a given maximum number of trajectories have been evaluated, the computer 40a may also display a full screen where the operator can modify one or more of the points of interest CPOI and / or modify the constraints for the optimization problem.
[0091] Consequently, in case the simulated movement corresponds to the requested movement (output "yes" from verification step 2212), the computer 40a proceeds to step 2214. However, in this case, apart from possible interactions required of the sensors 102 and / or actuators 104, the computer 40a transmits the program RPRG to the controller 30 which then executes the instruction sequence RPRG for controlling the movements of the robot arm 10.
[0092] In general, for the movement of the robot arm 10, a given command CMD is: · Only movement of the robot arm 10 along a selected trajectory without interaction with the sensors 12 / 102 and / or actuators 104; the movement of the robot arm 10 along a selected trajectory to reach a given final position and the subsequent interaction with the sensors 12 / 102 and / or actuators 104 once the robot arm has reached the final position; or Movement of the robot arm 10 along a selected trajectory and, in parallel therewith, interaction with the sensors 12 / 102 and / or actuators 104 may be required.
[0093] For example, to acquire an image of the surface of the component 20, the sequence of commands CMD may include a pair of commands, where a first command requests a movement at each point of interest and a second command requests acquisition of the image, so that in this case the movement of the robot arm 10 is interrupted in order to acquire the image.
[0094] Alternatively, a first command may request a movement to a given initial position and a second command may request a movement to a given final position in parallel with the acquisition of images. For example, for this purpose, the computer 40a may periodically monitor (via the controller 30) the position of the robot arm 10, in particular the TCP, and perform one or more movements identified via commands CMD, for example acquiring images when the robot arm 10 reaches a given position, for example when the robot arm reaches a given set point of a trajectory identified via instructions RPRG.
[0095] For example, in this case the camera needs to be kept at a certain distance from the surface of the component 20. Consequently, in various embodiments, in particular with regard to commands CMD that require interaction with sensors and / or actuators during the movement of the robot arm 10, the computer 40a may be configured to generate, in stage 1208, one or more constraints on the trajectory depending on the command CMD to be executed.
[0096] Once the interaction with the sensors 102 / 12, the actuators 104 / 14 and / or the controller 30 has been completed, the computer 40a then selects, in step 2216, a subsequent command CMD, possibly using for this purpose the field CON indicating the conditions for the execution of the command CMD.
[0097] Consequently, in step 2218, the computer 40a can check whether the executed command CMD was the last command of the program CPRG. In case there are one or more further commands CMD in the program CPRG (output "yes" from the verification step 2218), the computer 40a then returns to step 2210. Alternatively, in case the executed command CMD was the last command of the program CPRG (output "no" from the verification step 2218), the phase 2200 ends in a final step 2220.
[0098] Consequently, in the embodiment considered, the computer 40a automatically determines the position / alignment of the component 20 with respect to the robot arm 10 and, using the virtual environment (with the models CM and RM and possible further models of obstacles and / or the environment), automatically determines the trajectory that the robot arm 10 will follow and, in particular, executes the movements requested by the current command CMD of the program CPRG. Consequently, in this way, the operator does not have to manually generate the virtual environment 42 and, in particular, to position the component 20 with respect to the robot arm 10 and does not have to generate instructions for the controller 30 to execute the movements. Instead, the operator may simply use the graphic interfaces 42b and / or 42c that allow him to specify, via a graphic interface, possible points of interest CPOI (or other information that allows the automatic generation of points of interest CPOI) and then, by interacting for this purpose with the sensors 102, the actuators 104 and / or the controller 30, i.e. the robot arm 10, via the command CMD, to specify the macro operations that are then automatically executed by the computer 40a.
[0099] Naturally, without affecting the underlying principles of the invention, the details of the construction and embodiments may vary widely from those described and illustrated herein, merely by way of example, without thereby departing from the scope of the invention as defined in the appended claims. (Other possible items) (Item 1) A method for inspecting and / or handling a component (20) via a robotic arm (10), the movement of the robotic arm (10) being managed via a controller (30) in response to movement commands, the movement commands specifying one or more points of a trajectory in a reference system of the robotic arm (10), and wherein one or more sensors (102, 12) are mounted on the robotic arm (10) and / or located on a platform (14) on which the robotic arm (10) is mounted and / or located in an environment in which the robotic arm (10) is located, and wherein the controller (20) and the one or more sensors (102, 12) are connected to a computer (40a), the method comprising: receiving a three-dimensional model (CM) of the component (20); showing the three-dimensional model (CM) of the component (20) in a first virtual environment (42b) having a given reference system (REF), where the first virtual environment (42b) allows for designation of one or more points of interest (CPOI); receiving one or more points of interest (CPOI) in the reference frame (REF) of the first virtual environment (42b); presenting a graphic interface (42c; 42d) allowing to specify a sequence of commands (CPRG) comprising a plurality of commands (CMD) for interacting with said robotic arm (10) and / or said one or more sensors (102, 12), each command (CMD) comprising data identifying an action (AT) and one or more parameters (ARG) for said action (AT), wherein a first command (MOVE) requests a movement of said robotic arm (10) towards a given point of interest (CPOI1) of said one or more points of interest (CPOI); acquiring (2202) one or more images (IMG) and / or point clouds (PC) of the component (20) via one or more sensors (102, 12) and comparing the one or more images (IMG) and / or the point clouds (PC) with the three-dimensional model (CM) of the component (20) to determine a position of the component (20) relative to the robot arm (10); Transforming (2206) coordinates of the given point of interest (CPOI1) in the reference system (REF) of the first virtual environment (42b) into coordinates in the reference system of the robot arm (10) using the position of the component (20) relative to the robot arm (10); generating (2208) a second virtual environment (42a) using the three-dimensional model (CM) of the component (20) and the model of the robot arm (RM); For each command (CMD) of said sequence of commands (CPRG), the following steps: determining whether said command (CMD) corresponds to said first command (MOVE) requesting a movement of said robot arm (10) towards said given point of interest (CPOI1); selecting (2210) a trajectory for moving the robot arm (10) in the coordinates of the given point of interest (CPOI1) in the reference system of the robot arm (10) if the command (CMD) corresponds to the first command (MOVE), and evaluating the selected trajectory in the second virtual environment to determine whether the robot arm (10) can follow the selected trajectory without colliding with the three-dimensional model (CM) of the component (20); and generating one or more movement commands (RPRG) for the selected trajectory if the robot arm (10) can follow the selected trajectory without colliding with the three-dimensional model (CM) of the component (20) and sending the one or more movement commands (RPRG) to the controller (30). Repeating the step of A method for providing the above. (Item 2) acquiring (2202) via one or more sensors (102, 12) one or more images (IMG) and / or point clouds (PC) of one or more obstacles in the environment in which the robotic arm (10) is located; wherein the step of generating (2208) a second virtual environment comprises generating a three-dimensional model of the one or more obstacles in response to the one or more images (IMG) and / or the point cloud (PC) of the one or more obstacles in the environment, and placing the three-dimensional model of the one or more obstacles in the second virtual environment, wherein said evaluating the selected trajectory includes determining whether the robot arm (10) can follow the selected trajectory without colliding with the three-dimensional model of the one or more obstacles. The method according to item 1. (Item 3) The method according to item 1 or item 2, wherein the selection (2210) of a trajectory for moving the robot arm (10) in the coordinates of the given point of interest (CPOI1) in the reference system of the robot arm (10) includes defining an optimization problem by adding constraints on all the models present in the second virtual environment and on the coordinates of the given point of interest (CPOI1) as constraints, and selecting an optimized trajectory according to a given cost function, such as distance covered, inertia on joints, energy consumption, or a combination thereof. (Item 4) Said graphic interface (42c; 42d) makes it possible to specify a sequence of commands (CPRG): A graphic interface (42c) for specifying a list of commands (CMD); and / or a graphic interface (42d) for specifying said sequence of commands (CPRG) via a flow chart; The method according to any one of items 1 to 3, comprising: (Item 5) 5. The method according to any one of items 1 to 4, wherein the series of commands (CPRG) is saved in the form of a list, for example by means of an Excel, CSV or XML file. (Item 6) The second command (MOVE) is one or more: Movement at a point determined according to the previous command; Predetermined movements (Touch); and / or Opening a file containing a sequence of one or more motion commands and sending the one or more motion commands to the controller (30). 6. The method according to any one of items 1 to 5, further comprising: (Item 7) One or more third commands (Look, Listen) may be used to acquire data via one or more sensors (102, 12), e.g.: Obtaining the recording and storing said recording in a file; and / or taking a recording, generating a text through speech recognition of said recording, and comparing said recognized text with a reference text; and / or Acquiring an image and storing said image in a file; and / or Obtaining an image, generating text through a character recognition operation on the image, and comparing the recognized text with a reference text. 7. The method according to any one of items 1 to 6, wherein (Item 8) One or more actuators (104) are mounted on the robotic arm (10), where: one or more fourth commands (Reproduce, Touch) requesting activation of one or more of the one or more actuators (104); and / or one or more fifth commands (Tap, Ultrasound) simultaneously requesting activation of one or more of the one or more actuators (104) and acquisition of data via one or more sensors (102, 12); 8. The method according to any one of items 1 to 7. (Item 9) 9. The method according to any one of claims 1 to 8, wherein the robotic arm (10) is mounted on a mobile platform (14), such as an automated guided vehicle. (Item 10) 10. The method of any one of claims 1 to 9, wherein the sequence of commands (CPRG) includes data specifying, for each command (CMD), a respective condition (CON) indicating when the respective action (AT) should be performed. (Item 11) 11. The method according to any one of items 1 to 10, wherein each point of interest (CPOI) has a respective identifier and respective coordinates in the reference system (REF) of the first virtual environment (42b), wherein the graphic interface (42c; 42d) comprises a graphic interface (42c) for specifying a list of commands (CMD) and / or a graphic interface (42d) for specifying the sequence of commands (CPRG) via a flow chart, wherein each command (CMD) comprises data identifying an action (AT), a condition (CON) indicating when each of the actions (AT) should be executed and one or more parameters (ARG) for the action (AT), wherein a first command (MOVE) requests a movement of the robot arm (10) towards the given point of interest (CPOI1) of the one or more points of interest (CPOI) specifying as arguments the respective identifiers of the given points of interest (CPOI1). (Item 12) a robotic arm (10) and a controller (30), where movement of the robotic arm (10) is managed via the controller (20) in response to movement commands, where the movement commands specify one or more points of a trajectory in a reference frame of the robotic arm (10); one or more sensors (102, 12) mounted on the robotic arm (10) and / or located on a platform (14) on which the robotic arm (10) is mounted and / or located in the environment in which the robotic arm (10) is located; and a computer (40a), wherein the controller (20) and the one or more sensors (102, 12) are connected to the computer (40a), the computer (40a) being configured to implement the method according to any one of items 1 to 11; 1. A system for inspecting and / or handling a component, comprising: (Item 13) A computer program product loadable into the memory of at least one computer and comprising portions of software code for implementing the steps of the method according to any one of items 1 to 11.
Claims
1. 1. A method for inspecting and / or handling a component via a robotic arm, wherein movements of the robotic arm are managed via a controller in response to movement commands, wherein the movement commands specify one or more points of a trajectory in a reference system of the robotic arm, wherein one or more sensors are mounted on the robotic arm and / or located on a platform on which the robotic arm is mounted and / or located in an environment in which the robotic arm is located, wherein the controller and the one or more sensors are connected to a computer, the method comprising: receiving a three-dimensional model (CM) of the component; showing said three-dimensional model (CM) of said component in a first virtual environment having a given reference system (REF), said first virtual environment allowing to specify one or more points of interest (CPOI); receiving one or more points of interest (CPOI) in the reference frame (REF) of the first virtual environment; presenting a graphic interface allowing to specify a sequence of commands (CPRG) comprising a plurality of commands (CMD) for interacting with said robot arm and / or said one or more sensors, each command (CMD) comprising data identifying an action (AT) and one or more parameters (ARG) for said action (AT), wherein a first command (MOVE) requests a movement of said robot arm towards a given point of interest (CPOI1) of said one or more points of interest (CPOI); acquiring, via one or more sensors, one or more images (IMG) and / or point clouds (PC) of the component and comparing the one or more images (IMG) and / or the point clouds (PC) with the three-dimensional model (CM) of the component to determine the position of the component relative to the robot arm; transforming coordinates of the given point of interest (CPOI1) in the reference system (REF) of the first virtual environment into coordinates in the reference system of the robot arm using the position of the component relative to the robot arm; generating a second virtual environment using the three-dimensional model (CM) of the component and the model (RM) of the robot arm; For each command (CMD) of said sequence of commands (CPRG), the following steps: determining whether said command (CMD) corresponds to said first command (MOVE) requesting a movement of said robot arm to said given point of interest (CPOI1); selecting a trajectory for moving the robot arm in the coordinates of the given point of interest (CPOI1) in the reference system of the robot arm if the command (CMD) corresponds to the first command (MOVE), and evaluating the selected trajectory in the second virtual environment to determine whether the robot arm can follow the selected trajectory without colliding with the three-dimensional model (CM) of the component; and generating one or more motion commands (RPRG) for the selected trajectory if the robot arm can follow the selected trajectory without colliding with the three-dimensional model (CM) of the component and sending the one or more motion commands (RPRG) to the controller; Repeating the step of A method for providing the above.
2. acquiring, via one or more sensors, one or more images (IMG) and / or point clouds (PC) of one or more obstacles in the environment in which the robot arm is located; wherein the step of generating a second virtual environment comprises the steps of generating a three-dimensional model of the one or more obstacles in response to the one or more images (IMG) and / or the point cloud (PC) of the one or more obstacles in the environment, and placing the three-dimensional model of the one or more obstacles in the second virtual environment, wherein the evaluating the selected trajectory includes determining whether the robot arm can follow the selected trajectory without colliding with the three-dimensional model of the one or more obstacles. The method of claim 1.
3. The method of claim 1, wherein selecting a trajectory for moving the robot arm in the coordinates of the given point of interest (CPOI1) in the reference system of the robot arm comprises defining an optimization problem by adding constraints on all the models present in the second virtual environment and on the coordinates of the given point of interest (CPOI1) as constraints, and selecting an optimized trajectory according to a given cost function, such as distance covered, inertia on joints, energy consumption, or a combination thereof.
4. Said graphic interface makes it possible to specify a sequence of commands (CPRG): A graphic interface for specifying a list of commands (CMD); and / or A graphic interface for specifying said sequence of commands (CPRG) via a flow chart The method of claim 1 , comprising:
5. The method according to claim 1 , wherein said series of commands (CPRG) is saved in the form of a list, for example by means of an Excel, CSV or XML file.
6. The second command or commands (MOVE): Movement at a point determined according to the previous command; Predetermined movements (Touch); and / or Opening a file containing a series of one or more movement instructions and sending the one or more movement instructions to the controller. The method of claim 1 , further comprising:
7. One or more third commands (Look, Listen) may be used to obtain data via one or more sensors, e.g.: Obtaining a recording and storing said recording in a file; and / or taking a recording, generating a text through speech recognition of said recording, and comparing said recognized text with a reference text; and / or Acquiring an image and storing said image in a file; and / or Obtaining an image, generating text through a character recognition operation on the image, and comparing the recognized text with a reference text. The method of claim 1 , further comprising:
8. One or more actuators are mounted on the robotic arm, where: one or more fourth commands (Reproduce, Touch) requesting activation of one or more of the one or more actuators; and / or one or more fifth commands (Tap, Ultrasound) simultaneously requesting the actuation of one or more of the one or more actuators and the acquisition of data via one or more sensors; The method of claim 1.
9. The method of claim 1 , wherein the robotic arm is mounted on a mobile platform, such as an automated guided vehicle.
10. 2. The method of claim 1, wherein the sequence of commands (CPRG) includes data specifying, for each command (CMD), a respective condition (CON) indicating when the respective action (AT) should be performed.
11. 2. The method of claim 1, wherein each point of interest (CPOI) has a respective identifier and respective coordinates in the reference system (REF) of the first virtual environment, and wherein the graphic interface includes a graphic interface for specifying a list of commands (CMD) and / or a graphic interface for specifying the sequence of commands (CPRG) via a flow chart, wherein each command (CMD) includes data identifying an action (AT), a condition (CON) indicating when each of the actions (AT) needs to be executed, and one or more parameters (ARG) for the action (AT), and wherein a first command (MOVE) requests movement of the robot arm to the given point of interest (CPOI1) of the one or more points of interest (CPOI) specifying the respective identifiers of the given points of interest (CPOI1) as arguments.
12. a robotic arm and a controller, where movement of the robotic arm is managed via the controller in response to movement commands, where the movement commands specify one or more points of a trajectory in a reference frame of the robotic arm; one or more sensors mounted on the robotic arm and / or located on a platform on which the robotic arm is mounted and / or located in the environment in which the robotic arm is located; and A computer, wherein the controller and the one or more sensors are connected to the computer, the computer being configured to implement the method according to any one of claims 1 to 11.
1. A system for inspecting and / or handling a component, comprising:
13. A computer program loadable into the memory of at least one computer and comprising portions of software code for implementing the steps of the method according to any one of claims 1 to 11.