A method for automatically performing operations on an object using a tool held by a multi-joint system, and equipment for implementing said method.

Abstract

The present invention relates to a method and apparatus for automatically performing an operation on an object (2) using a tool (4) held by an articulated system (5) operable in a working environment (3) under conditions where the object (2) and the working environment (3) are variable or insufficiently defined to perform the operation. According to the invention, the method comprises at least the following steps: A) capturing images of the object (2) and the working environment (3) in the form of a scatter plot using at least one 3D sensor (6), merging said images with a CAD model of the articulated system and a possible CAD model of the environment to form a workpiece image and defining collision avoidance parameters; B) defining a path of the tool (4) on a portion of the workpiece image representing the object (2) and simulating corresponding movements of the articulated system (5) and the tool on the workpiece image to ensure that the operation is feasible; and C) actually performing movements of the articulated system (5) holding the tool (4) according to the defined path (10) to perform the operation on the object (2) if step B determines that the operation is feasible.

Description

Technical Field

[0001] The present invention relates to the technical field of performing operations such as positioning on an object within a working environment, where the position and geometric shape of the object and its working environment may change, and / or the definitions thereof are insufficient for performing operations on the object, such as cutting, welding, marking, peeling, painting, surface treatment, sensors, and other various types of analysis tools.

[0002] For example, the present invention is applied to, but not limited to, the disassembly technical field where operations such as measurement, cutting, and gripping of objects that may have risks related to radioactivity are performed.

[0003] More specifically, the present invention relates to a method and equipment for automatically performing operations on an object using a tool that is held by a multi-joint system and movable within a working environment, in a situation where the position and geometric shape of the object and the working environment are changing, and / or the definitions thereof are insufficient for performing the operations in an automatic mode. The operations may be of any type, performed on any type of object, using any type of tool, under any type of environment.

[0004] The present invention can be advantageously applied, for example, when performing some operations on an object in a dangerous and enclosed environment where an operator cannot access or move freely within it.

Background Art

[0005] In the field of the present invention, for example, in the field of disassembly of radioactive objects, the disassembly operation can be manually performed by an operator in a closed environment.

[0006] Dismantling operations include measuring, cutting, and gripping the object. In particular, the purpose of cutting operations is to optimize the filling of waste containers by generating smaller components. Therefore, these smaller parts need to be handled by the system and placed into the containers.

[0007] Each of these operations is performed by a worker equipped with tools such as an angle grinder or any other type of tool that enables the operation, as well as restraining protective gear (such as a breathable coverall and safety gloves) that takes into account the radiation environment of the work site and the nature of the operation being performed.

[0008] Therefore, performing such operations is not optimal in terms of worker safety, difficulty, and time required.

[0009] Otherwise, the operation cannot be performed by the operator in contact with the tool (for measurement, cutting, and gripping), and the tool is held by a multi-joint system that performs the operation. The operator remotely drives the system using a control horn, while viewing indirect images from a camera and screen. Disadvantages of such systems include high cost, low precision in controlling speed, force, and positioning, long completion times, and rapid wear of tool consumables (blades, discs). [Overview of the project] [Problems that the invention aims to solve]

[0010] The present invention aims to compensate for the shortcomings of the prior art by providing a method and equipment for performing operations on objects within a work environment, thereby ensuring optimal conditions regarding safety, accuracy, and speed.

[0011] For this purpose, a method was developed to automatically perform operations on an object using a tool held by a movable, articulated system within the work environment. [Means for solving the problem]

[0012] According to the present invention, the method ○ Capture images of all or part of the object and work environment in the form of an overall point cloud using at least one 3D sensor. ○ Merge that image with all or part of the existing CAD models of the multi-joint system and the as-built environment, thereby producing the work image. ○ Define collision avoidance parameters Step A and - Step B involves (preferably by the operator) defining the tool's trajectory for a portion of the work image representing the object, and simulating the corresponding movement of the articulated system and tool on the work image to ensure that the operation is feasible. - If the operation is deemed feasible in step B, then step C involves actually performing the operation of the articulated system to hold the tool according to a defined trajectory in order to perform the operation on the object. It includes at least [this].

[0013] In the present invention, a 3D sensor means any means of digitizing an object or a part of an object into the form of a three-dimensional image, and this includes, in particular, 3D cameras, laser scanners, video means, webcams, and the like.

[0014] Collision avoidance parameters include, for example, a threshold distance beyond which safety functions such as a complete halt or deceleration of all movement of the articulated system and tools are automatically activated, which in turn constitutes a collision avoidance parameter.

[0015] In this way, the present invention makes it possible to perform various operations that were initially uncertain, unknown, or inaccessible in an environment remotely and in an automated mode, while ensuring that they are performed correctly.

[0016] The present invention is advantageous in that the work environment is digitized in the form of a total point cloud prior to any movement of the articulated system. This allows the location of various obstacles to the movement of the articulated system, i.e., the locations of various elements in the work environment and the objects to be manipulated, to be determined. Therefore, collisions between the articulated system or its tools and various elements in the environment can be avoided by collision prevention parameters.

[0017] This is particularly advantageous when the operator controlling the method being implemented cannot directly see the work environment. The operator visualizes the work environment and objects as a point cloud on a display and defines the trajectory of the tool to perform the desired operation, such as a cutting operation.

[0018] The present invention is advantageous in that, by employing this method, it becomes possible to simulate the corresponding movements of a multi-joint system and its tools, particularly in CAD simulations, and enables them to perform movements without causing collisions.

[0019] Therefore, if the feasibility of the operation is confirmed through simulation, the operation of the multi-joint system that holds the tool can be actually performed to carry out the operation on the target object accordingly.

[0020] This method can be implemented in at least three steps. That is, - Step A: The environment and objects are determined by digitizing the entire point cloud. - Step B defines and simulates the trajectory, and - Step C to execute.

[0021] Preferably, this method controls the operation of the articulated system to capture a detailed image of the zone of the object in the form of a point cloud with higher density and higher precision than the overall point cloud using a 3D sensor held by the articulated system, and further includes step A' between step A and step B of incorporating the captured image of the corresponding zone of the object into the work image instead of the corresponding part by the overall point cloud.

[0022] This additional step provides a more highly accurate and improved work image for defining various trajectories of the operations to be performed. Further, this additional step enables the capture of images of zones that are not visible by the sensor used in step A by the operation of the articulated system, thereby making the work image complete.

[0023] The 3D sensor held by the articulated system can either remain permanently attached to the articulated system or be stored in its vicinity. In the latter configuration, the method includes step A'' between step A and step A' of automatically operating the articulated system to grip the 3D sensor stored in the vicinity and connect it to the 3D sensor.

[0024] According to a specific embodiment, the method includes step C' between step B and step C of automatically operating the articulated system to optionally remove and lower the 3D sensor and connect it to a tool stored in the vicinity.

[0025] This feature enables storing a plurality of tools in the vicinity so that they can be connected to the articulated system.

[0026] If several different tools are made available, the method further includes step B' before step B of selecting, in the simulation of step B, the tool for performing the operation from within the tool database for the simulation.

[0027] Advantageously, and also to prevent tool damage, the method is superior in that the operating speed of the articulated system that holds the tool in direct contact with the object in step C can be adjusted in real time as needed based on direct or indirect measurement of the force applied to the tool.

[0028] To ensure safe operation of the articulated system holding the 3D sensor in step A', the articulated system automatically slows down when the 3D sensor or part of the articulated system approaches an object or element of the work environment, and automatically stops when the 3D sensor or part of the articulated system reaches a safe distance from the object or element of the work environment according to the collision avoidance parameters defined in step A.

[0029] The definition of the tool's trajectory for the portion of the work image representing the object in step B consists of positioning either at least one start-end pair or at least one predefined geometric shape, selected from a library consisting of at least one line, plane, mask, etc., within the work image.

[0030] The present invention relates to an apparatus for implementing the method described above, comprising a multi-joint system, a computer processing system and at least one 3D sensor connected to a display, - Based on images captured by at least one 3D sensor, the entire point cloud of all or part of the object and work environment is displayed on the screen. - To represent a work image obtained by merging the overall point cloud, an existing CAD model of the articulated system, and all or part of the existing CAD models of the as-built environment. Regarding equipment equipped with a designed 3D sensor, The computer processing system also enables the definition of tool trajectories relative to a portion of a work image representing an object, simulates the corresponding movements of the articulated system and tool on the work image to ensure that the operation is feasible, and, if the operation is deemed feasible, is configured to actually operate the articulated system, which holds the tool according to the defined trajectory for performing that operation on the object.

[0031] Advantageously, the equipment includes means for adjusting in real time the operating speed of the articulated system that holds the tool in direct contact with the object in step C, based on direct or indirect measurement of the force applied to the tool.

[0032] Other features and advantages of the present invention will become clearer, without limitation, from the following description, with reference to the accompanying drawings. [Brief explanation of the drawing]

[0033] [Figure 1] This is a schematic perspective view of equipment according to the present invention, showing a multi-joint system holding a tool about to perform an operation on an object in its work environment. [Figure 2] This is a schematic diagram similar to Figure 1, showing a multi-joint system holding a 3D camera. [Figure 3] This perspective view is displayed on a screen as a work image created by merging the overall point cloud of the object and parts of the work environment obtained from images captured by one 3D camera with the CAD model of the multi-joint system from the other camera. [Figure 4] This figure is similar to Figure 3, but shows a multi-joint system holding a 3D camera for capturing detailed images of an object, with those details of the object incorporated into the workpiece image in place of the corresponding parts of the overall point cloud. [Figure 5] This diagram is similar to Figure 4, but the tool's trajectory is defined in the work image portion representing the object. [Figure 6]This is a schematic diagram similar to Figure 5, illustrating a simulation of the operation of a multi-joint system and tool in a workpiece image. [Figure 7] This diagram shows in detail the positioning of the plane that intersects with the portion of the workpiece image representing the object. [Figure 8] This diagram is similar to Figure 7, and shows how the motion trajectory of a multi-joint system is automatically calculated by automatically calculating the trajectory obtained from the intersection of a positioned plane and a portion of the workpiece image representing the object. [Figure 9] This diagram illustrates the simulation stages of the operation of a multi-joint system. [Figure 10] This figure shows a simplified flowchart of the method according to the present invention. [Modes for carrying out the invention]

[0034] The present invention relates to a method and equipment (1) for automatically performing operations on an object (2) placed in a working environment (3).

[0035] The present invention is not limited to any single operation, but may relate to operations such as measurement, cutting, gripping, welding, writing, marking, peeling, painting, surface treatment, positioning of sensors and all other types of analytical tools. Such operations are performed by a tool (4) held by a movable articulated system (5) within a working environment (3).

[0036] The object (2) on which the operation by the method of the present invention is performed can be a radioactive object, any other object to be dismantled, or any kind of repair or welding.

[0037] The work environment (3) related to the object (2) can be of any kind, such as a dangerous and enclosed work environment, a radioactive environment, or an environment where workers cannot access or move freely, such as working at height.

[0038] As shown in Figure 1, the equipment (1) comprises a multi-joint system (5) in the form of a robotic arm that is movable in all directions within the work environment (3). The equipment (1) comprises at least one 3D sensor, such as a 3D camera (6A, 6B, 6C), to capture and digitize images of all or part of the object (2) and the work environment (3) and to render a three-dimensional representation of the entire point cloud (7) of all or part of the object (2) and the work environment (3) on a known computer processing system and display.

[0039] In the illustrated example, the equipment (1) comprises three 3D cameras (6A, 6B, 6C) positioned and fixed around and over an arch (8) surrounding a multi-joint system (5).

[0040] As explained in relation to Figure 3, in order to safely perform operations on the object (2), the equipment (1) implements a method that includes at least one step A, which involves capturing images of all or part of the object (2) and the work environment (3) in the form of a whole point cloud (7) using 3D cameras (6A, 6B, 6C) mounted on the arch (8), merging the whole point cloud (7) with an existing CAD model of the articulated system (5) and an existing CAD model of the work environment, such as the CAD model of the arch (8), and displaying the resulting work image (17) on a display (assembly in Figure 3).

[0041] This allows for the identification of potential obstacles in the work environment (3) and ensures the safe operation of the articulated system (5). It should be noted that all or some of the standard elements of the work environment (3), such as the arch (8), 3D cameras (6A, 6B, 6C), the articulated system (5) itself, and its tools (4), may be known and already modeled in the CAD.

[0042] In this way, the operation of the articulated system (5) and its tools (4) can be known in relation to various elements of the work environment (3), and collisions can be automatically avoided by predefined collision avoidance parameters.

[0043] In order to perform a desired operation on the object (2), the equipment (1) is preferably equipped with various types of tools (4) and 3D sensors such as a 3D camera (16), which are stored nearby, for example, in a dedicated storage box.

[0044] This 3D camera (16) may be of the same type as the 3D cameras (6A, 6B, 6C) or of a different type, but in the latter case, it is advantageous that it is considered to have higher accuracy than the 3D cameras (6A, 6B, 6C).

[0045] Therefore, it is advantageous that the method according to the present invention includes, after step A'', step A'' which involves automatically and safely operating the articulated system (5) to send it to a nearby stored 3D camera (16), grasping the 3D camera (16), and connecting to it.

[0046] After the articulated system (5) grasps the 3D camera (16), the method includes step A' (see Figure 2) in which the movement of the articulated system (5) is controlled to capture detailed images of various zones of the object (2) using the 3D camera (16), which has higher precision than the 3D cameras (6A, 6B, 6C), in the form of a point cloud with higher density and precision than the overall point cloud (7), and incorporates the captured images (9) of the corresponding zones of the object (2) into the workpiece image (17) (see Figure 4) in place of the corresponding parts of the overall point cloud (7).

[0047] The operation of the articulated system (5), which captures various images, is controlled either automatically or remotely by an operator in the work environment (3), for example, by a control lever, to enable the capture of images of inaccessible zones, or by directly selecting a specific zone from the overall point cloud (7), which triggers the automatic operation of the articulated system. In this configuration, the operator selects the zone in which they wish to improve the accuracy of the object modeling. The software then calculates the position of the articulated system (5) required to capture images from the best viewpoint. It should be noted that the operation of the articulated system (5) is performed without collisions between the articulated system (5), the 3D camera (16), the object (2), and the various elements of the work environment (3), because everything is modeled in the work image (17) by CAD, through the overall point cloud (7), or through detail images (9).

[0048] Furthermore, to enhance the operational safety of the equipment (1), in step A', when a part of a tool (4) or articulated system (5) whose CAD modeling results are known approaches an object (2) or an element of the work environment (3), the movement of the articulated system (5) automatically slows down, and the movement stops when the tool (4) or part of the articulated system (5) reaches a safe distance from the obstacle.

[0049] As described above, the operator controlling the movement of the articulated system (5), particularly the movement of the held 3D camera (16), captures detailed images (9) of the target zone that they intend to manipulate. These detailed images (9) of the object (2) are automatically incorporated into the workpiece image (17), thereby allowing for a partially more accurate reconstruction of the object (2) and providing detailed information about its geometric shape at the location where the manipulation takes place.

[0050] Assuming that several different types of tools (4) are available, the method includes step B' of selecting a tool (4) for the simulation to perform the operation from a tool database for the simulation.

[0051] In that case, referring to Figure 5, the method includes step B, which defines the trajectory (10) of the tool (4) for the portion of the work image (17) representing the object (2), and simulates the operation of the corresponding articulated system (5) and tool (4) on the work image (17) to ensure that the operation is feasible in terms of direction, accessibility, and lack of collisions (see Figure 6).

[0052] By capturing detailed images (9) of the object (2) and incorporating them into the workpiece image (17), the operator can obtain more precise assistance in defining and positioning the tool's (4) trajectory (10).

[0053] To define the trajectory (10), the operator can position start-end pairs or geometric shapes selected from a library consisting of lines, planes, masks, etc., on a work image (17) displayed on the screen, and the computer processing system is designed to automatically calculate the trajectory (10) on the object (2). If necessary, the processing system can manually adjust the trajectory (10), or the operator can directly draw the lines of the trajectory (10) on the representation of the object (2).

[0054] For example, in Figure 5, a plane (11) is positioned relative to the surface representation of the object (2) to be cut, and the computer processing system defines the trajectory (10) by the intersection of the plane (11) and its surface. This method is also shown in Figure 7, where the positioning of the plane (11) can be seen, and in Figure 8, it can be seen that the trajectory (10) is calculated and plotted at the intersection of the plane (11) and the representation of the object (2).

[0055] Once the trajectory (10) is calculated, the computer processing system allows the corresponding articulated system (5) and tool (4) to be simulated within the workpiece image (17) to enable operation, provided that this operation is feasible for the tested trajectory (10), tool (4), and articulated system (5). The steps of the operation test are shown, for example, in Figures 6 and 9. If the simulation results show that operation is possible, that is, possible in terms of direction and access, and that the operation will not cause collisions between the various elements of the equipment (1) and the work environment (3), then the operation can be performed. Step B is performed as many times as necessary to obtain a feasible operation.

[0056] In this case, it is advantageous for the method to include step C', which involves removing the 3D camera (16) and placing it in a dedicated storage box or the like, and then automatically operating the articulated system (5) to connect it to a pre-selected tool (4) that is also stored nearby.

[0057] After the tool (4) is connected, the method includes step C, if the simulation has shown that the operation is feasible, to actually operate the articulated system (5) that holds the tool (4) according to the defined and validated trajectory and perform the operation on the object (2).

[0058] Preferably, in step C, the equipment includes means for adjusting in real time the operating speed of the articulated system (5) that holds the tool (4) in direct contact with the object (2) in step C, based on direct or indirect measurement of the force acting on the tool, in order to prevent damage to the tool (4). For example, this measurement can be obtained by measuring the current consumed by the tool (4) or motor to move the articulated system (5), or by a force sensor placed between the articulated system (5) and the tool (4).

[0059] To make it easy to understand, Figure 10 shows the sequence of steps in the method in the form of a simplified flowchart.

[0060] Advantageously, when it is necessary to perform multiple operations sequentially on the object (2), such as multiple cutting operations, the subsequent cutting trajectory (10) is defined by a mask time between the previous cutting operations, or the multiple trajectories (10) are simulated and recorded so that they are performed sequentially in an order that can be modified by the operator.

[0061] The present invention is also particularly advantageous in cases related to cutting an object (2), for example, under changing environments due to new elements or potential obstacles resulting from the addition of objects or the creation of spaces, in which case, by repeating steps A through C of the method of the present invention, up-to-date images and simulations for continuing operations on the object (2) can be easily obtained without the risk of collision.

[0062] The display shows various elements of the equipment (1), object (2), and environment in CAD mode. Preferably, the articulated system (5) is represented by interactive colors. That is, the articulated system (5) is represented by green, for example, and as one part of it approaches an obstacle, the color of that part changes to orange when the articulated system (5) and / or tool (4) enter a collision risk zone defined by the collision management parameters, and then changes to red when the articulated system (5) and / or tool (4) reach a threshold distance defined by the collision management parameters and stop moving.

[0063] From the above, it is clear that the present invention provides a method and equipment (1) for automatically and safely operating on an object (2) located in a work environment (3) where the position and geometric shape of the object (2) and the work environment change and / or are not sufficiently defined for operation.

[0064] This method can be adapted to any type of geometric shape or property of the object (2). The first step, which involves digitizing the environment and object (2) to define the overall point cloud, is performed using a 3D camera, which also enables adaptation to any type of position and geometric shape of various elements of the object (2) and environment. Processing time is short, handling 500,000 measurement points per second. The details obtained from the environment are important, and the information is continuous. Therefore, realistic rendering and modeling can be performed in real time, and the operator can easily visually check whether the reconstruction in the point cloud (7) is correct. [Explanation of symbols]

[0065] 1 equipment 2. Object 3. Working Environment 4 Tools 5. Multi-joint system 6. 3D cameras, 3D sensors 7 point cloud 8 Arches 9 images 10 orbits 11 plane 16 3D cameras, 3D sensors 17 Work Images

Claims

1. In a method for automatically performing operations on an object (2) using a tool (4) held by a movable multi-joint system (5) within a work environment (3), - Step A involves capturing images of all or part of the object (2) and the work environment (3) in the form of a whole point cloud (7) using at least one 3D sensor (6A, 6B, 6C), merging the whole point cloud (7) with an existing CAD model of the articulated system (5) and possible existing CAD models of all or part of the completed work environment (3), thereby obtaining a work image (17), and further defining collision prevention parameters. - Step B, which involves defining the trajectory (10) of the tool (4) with respect to the portion of the work image representing the object (2), and simulating the corresponding movements of the articulated system (5) and the tool in the work image (17) to ensure that the operation is executable. - If the operation is deemed feasible in step B, then step C is to actually perform the movement of the articulated system (5) that holds the tool (4) according to the trajectory (10) defined for performing the operation on the object (2). It includes at least, A method characterized by including step A' between step A and step B, which involves controlling the operation of the articulated system (5) to capture detailed images (9) of zones of the object (2) using a 3D sensor (16) held by the articulated system (5) in the form of a point cloud with higher density and higher accuracy than the overall point cloud, and incorporating the detailed images (9) of the corresponding zones of the object (2) into the workpiece image (17) in place of the corresponding portion of the overall point cloud (7).

2. The method according to claim 1, characterized in that step A'', between step A and step A', is to automatically operate the articulated system (5) to grasp and connect the 3D sensor (16) stored nearby.

3. The method according to claim 1, characterized in that step C' is included between step B and step C, in which the articulated system (5) is automatically operated to arbitrarily remove and lower the 3D sensor (16) and connect it to the tool (4) stored nearby.

4. The method according to claim 1, characterized in that a tool (4) for performing the above operation is included in the simulation of step B, and step B' for selecting a tool from a tool database for the simulation is included before step B.

5. The method according to claim 1, characterized in that in step C, the operating speed of the articulated system (5) that holds the tool (4) in direct contact with the object (2) is adjusted in real time based on direct or indirect measurement of the force applied to the tool (4).

6. The method according to claim 1, characterized in that in step A', when the 3D sensor (16) or a part of the articulated system (5) approaches an object (2) or an element of the work environment (3), the movement of the articulated system is automatically slowed down, and when the 3D sensor (16) or a part of the articulated system (5) reaches a safe distance from the object (2) or an element of the work environment (3) according to the collision prevention parameter defined in step A, the movement is automatically stopped.

7. The method according to claim 1, characterized in that the trajectory of the tool (4) with respect to the portion of the work image representing the object (2) in step B is defined by positioning at least one start-end pair or at least one predefined geometric shape, in particular a plane (11), at least one line, or a mask within the work image (17).

8. An apparatus (1) for implementing the method according to claim 1, comprising a multi-joint system (5), a computer processing system and at least one 3D sensor (6) connected to a display, wherein the computer processing system - Based on images captured by at least one 3D sensor (6), the entire point cloud (7) of all or part of the object (2) and the work environment (3) is displayed on the display. - Represents a work image (17) obtained by merging the overall point cloud (7) of the existing CAD model of the articulated system (5) and all or part of the existing CAD models of the work environment (3), - Enables the definition of the tool's (4) trajectory (10) relative to the portion of the work image (17) representing the object (2), - The corresponding movements of the articulated system (5) and the tool (4) are simulated using the work image (17) to ensure that the operation is executable. - The apparatus is characterized in that, if the operation is deemed feasible, it is designed to actually operate the articulated system (5) that holds the tool (4) according to the trajectory (10) defined for performing the operation on the object (2).

9. The apparatus according to claim 8, characterized in that it includes means for adjusting in real time the operating speed of the articulated system (5) that holds the tool (4) in direct contact with the object (2) in step C, based on direct or indirect measurement of the force applied to the tool (4).

Images