Apparatus, system, and method for 3D scanning of objects in an automated disassembly process

Abstract

Aspects concern an apparatus (100) for generating a 3D scan (160) of an object (110) in an automated disassembly process, the apparatus (100) comprising a 3D scanner (120), the 3D scanner (120) configured to mounted on a machine (130), the machine (130) configured to move in at least one rotational motion and at least one translational displacement; a platform (140) positioned at a pre-determined distance from the machine (130), the platform (140) rotatable about a centre (140A); and a processor (150) arranged in data communication with the machine (130) and the platform (140), the processor (150) configured to obtain a set of scan parameters as input and provide a set of control data as output, the set of control data comprising at least one of a first control data to control the machine (130) to move the machine (130) to a desired pose of the 3D scanner, and a second control data to control the platform (140) to move the platform (140) to expose a surface of the object (110) within optical alignment of the 3D scanner.

Description

APPARATUS, SYSTEM, AND METHOD FOR 3D SCANNING OF OBJECTS IN AN AUTOMATED DISASSEMBLY PROCESSTECHNICAL FIELD

[0001] This disclosure relates to an apparatus, a system and a method for 3D scanning of objects in an automated disassembly process. The disclosure is particularly suited, but not limited to 3D scanning for purpose of identification of a product type to retrieve necessary product information for disassembly.BACKGROUND

[0002] In the recent years, increasing amount of e-waste was generated but a relatively small percentage of such e-waste have been collected for recycling. Under the current linear economy model (make-take-waste model), a rising influx of end-of-life (EoL) products are being disposed. Current recycling facilities typically use traditional recycling technologies without proper pre-treatment, for example, shredding, crushing and magnetic sorting for mass recovery. These traditional recycling technologies have some disadvantages including relatively low recovery rate and relatively energy -intensive. In addition, some of these recycling technologies may pose health hazards.

[0003] As more linear economy model has to be transformed to the circular economy model, and with used products being returned to the life circle via recycling, reuse, repair and remanufacturing, these EoL products require an automated pre-treatment process in such a way that components of EoL products are disassembled, diagnosed, and / or sorted.

[0004] 3D scanning is playing an important role in automated pre-treatment process and is being used for automated disassembly, typically in the identification of a product type to retrieve necessary product information for disassembly. Such product information may be generated as computer-aided design (CAD) files or bill of materials (BOM) files. The product information may include one or more levels in disassembly, disassembly instructions and / or disassembly order, etc.

[0005] The files generated from the 3D scanning may also be used to identify or locate components so as to create a full representation 3D model of an object. In addition, 3D scanning may be used for planning pre-treatment process, for example, 3D scanning can be used to create a plan for disassembly.

[0006] Existing 3D scanning solutions may comprise a variety of disadvantages for automated 3D scanning in the pre-treatment process. Firstly, existing solutions are not able to scan objects of different sizes, as their scanning patterns are fixed and not able to adapt to objects of greatly different sizes. Secondly, the existing solutions do not provide users with the flexibility to tweak the scan parameters, which would enable the user to optimize the scanning process for different use cases, to choose quality of scan over time and vice versa. Thirdly, existing solutions are not adequately equipped for scanning different views of objects.

[0007] There exists a need to provide an improved solution for 3D scanning in an automated disassembly process.SUMMARY

[0008] This disclosure was conceptualized to tackle the limitations of application of automated 3D scanning in an automated disassembly process.

[0009] A 3D scanning process that uses a 3D scanner that is mounted on a robotic arm, for example an industrial or collaborative robotic arm, as well as an automated turntable is contemplated. In some embodiments, the 3D scanner may be a structured light 3D scanner. In some embodiments, the robotic arm may be a robotic arm having at least six degree-of-freedom (DOF). The 3D scanner may be configured to obtain a fully automated scan of an object, is adaptive, compact, and user controllable.

[0010] According to the present disclosure, an apparatus as claimed in claim 1 is provided. A computer-assisted method according to the invention is defined in claim 8. A computer program comprising instructions to execute the computer-assisted method is defined in claim 14.

[0011] The dependent claims define some examples associated with the apparatus and method, respectively.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The invention will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which:- FIG. 1 is a schematic diagram of an embodiment of an apparatus for generating 3D scans of objects in an automated disassembly process;- FIG. 2A is a flow diagram of a method of operating the apparatus in scanning an object (scanning workflow);- FIG. 2B is an illustration of an irregularly shaped object 110 for scanning and the maximum dimensions associated with the object 110;- FIG. 3 is a schematic diagram showing the calculation of scanning angles and maximum scanning angles of the apparatus;- FIG. 4 is a schematic diagram showing a 3D scanner (camera) pose calculation according to some embodiments;- FIG. 5 is a flow diagram depicting a communication process between a robot and scanner; and- FIG. 6 is a flow chart of a method for 3D scanning of objects in an automated disassembly process according to some embodiments.DETAILED DESCRIPTION

[0013] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized and structural, and logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0014] Embodiments described in the context of one of the systems or methods are analogously valid for the other systems or methods.

[0015] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0016] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0017] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0018] As used herein, the term “data” may be understood to include information in any suitable analog or digital form, for example, provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. The term data, however, is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.

[0019] As used herein, the term “optically aligned” or “aligned optically” refers to the positioning and alignment of the various components of an apparatus / system so as to enable the apparatus / system to operate as intended. For example, a structured light 3D scanner projects electromagnetic radiation, which may be in the form of a series of parallel stripe patterns, onto a surface of the target object that is being scanned. The surface shape of the object distorts the stripe pattern. The pattern is then recorded by image capturing devices, such as cameras, on the 3D scanner, and the distortion allows a processor installed with image processing software to calculate the 3D model.

[0020] As used herein, the term “module” refers to, or forms part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may include memory (shared, dedicated, or group) that stores code executed by the processor.

[0021] As used herein, the terms ‘first’, ‘second’, ‘third’, and so on, are used for purposes of clarity and do not imply order or precedence.

[0022] As used herein, the term “pre-treatment process” is in a context of wastemanagement, and may include a disassembly process.

[0023] As used herein, the term “disassembly process” refers to a process whereby an object is separated into its components. In some embodiments, the disassembly process may be used to separate a product into its components and / or subassemblies via non-destructive or semidestructive processes which target the connectors / fasteners. Such disassembly process may have application in various industries, in particular, but not limited to, a waste treatment facility, reverse engineering processes, etc. The disassembly process may broadly include aguided disassembly process using augmented reality, virtual reality, and / or mixed reality; an automated disassembly (i.e. full automation).

[0024] As used herein, the term “3D scanner” may refer to one or more image capturing devices, and may include various 3D scanning techniques. In some embodiments, the 3D scanners such as structured light 3D scanners, which can be configured to collect high quality data. In some embodiments, the 3D scanning techniques may be based on stereo vision methods, and / or time-of-flight methods.

[0025] It is envisaged that structured light 3D scanners, may produce scanning having relatively higher precision and / or accuracy. A structured light 3D scanner may be configured to work using principles of triangulation. A light source of the 3D scanner may project a fringe pattern across the scan target object surface, and two cameras of the 3D scanner may be used to capture the surface geometry based on the pattern distortion, calculating 3D coordinate measurements. The 3D scanner may process coordinate data into a "point cloud," creating a digital image of the object. In some embodiments, the light source of the 3D scanners may be configured to emit white light. In some embodiments, the light source may be configured to emit other light such as blue light, which may be used to capture data on shinier and darker colored surfaces and filters out the ambient light.

[0026] As used herein, the term “processor” refers to a circuit, including analog circuits or components, digital circuits or components, or hybrid circuits or components. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a "circuit" in accordance with an alternative embodiment. A digital circuit may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, or a firmware.

[0027] As used herein, the term “obtain”, as used herein, refers to the processor which actively obtains the inputs, or passively receives inputs from a user interface and / or one or more sensors. The term obtain may also refer to the processor which receives or obtains inputs from a communication interface, e.g. the user interface. The processor or the dosing module may also receive or obtain the inputs via a memory, a register, and / or an analog-to-digital port.

[0028] An embodiment of the disclosure is shown in FIG. 1, which illustrates a setup of an apparatus 100 for generating one or more images, in the form of three-dimensional (3D) scans 160, of the object 110. The apparatus 100 may be configured for generating 3D scans 160 of an object 110 in an automated disassembly process. In some embodiments, the automateddisassembly process may form part of a waste or materials recovery facility, such as an e-waste recovery facility.

[0029] The apparatus 100 may comprise a 3D scanner 120, the 3D scanner 120 configured to mounted on a machine 130, the machine 130 configured to move in at least one rotational motion and at least one translational displacement. In some embodiments, the 3D scanner may be a structured light 3D scanner.

[0030] In some embodiments, a platform 140 may be positioned at a pre-determined distance from the machine 130, and the platform 140 may be rotatable about a centre 140A.

[0031] In some embodiments, the object 110 may be freely positioned on the platform 140. The distance between object 110 and the 3D scanner 120 or machine 130 may then be measured via a software function of the 3D scanner 120 or machine 130. In some embodiments, distance sensors (ultrasonic sensors, laser sensors, etc.) may be positioned to obtain distance measurements between the object 110 and the machine 130 and / or the 3D scanner 120.

[0032] In some embodiments, the machine 130 may be a robotic arm with six degree of freedom. In some embodiments, the robotic arm may be a six-axis articulated robotic arm, and may comprise one or more controllers for communication with a processor 150. It is appreciable that the robotic arm may have higher degrees of freedom, i.e. more than 6 degrees of freedom.

[0033] In some embodiments, the robotic arm may be a collaborative robot (cobot), the cobot being intended for direct human-robot interaction within a shared space or shared area. Such cobot may be especially suited in situations where human users / controllers and the robots are in close proximity. The cobot may be formed from lightweight construction materials, rounded edges, and may comprise a limitation of speed and force to enhance safety during operation.

[0034] In some embodiments, the robotic arm may comprise sensors (e.g. laser displacement sensors) and software that enhance safe behaviour during usage. For example, the robotic arm may be configured to stop operation when it detects an obstacle (e.g. another robot, human user) in its movement path. In some embodiments, the robotic arm may be equipped with additional components such as force sensors, torque sensors, tool changer, a customized mounting frame for mounting the 3D scanner 120 and other robot locomotion, such as unbolting components / sensors, diagnosis components, etc.

[0035] The platform 140 may comprise a turntable mounted on a fixture to support the object 110. In operation, the turntable may be configured to be rotated about a centre 140A of the fixture so as to expose or align a part of the object 110 to the 3D scanner 120 for the 3D scan. The exposure may correspond to the optical alignment of the 3D scanner 120 to a surface of the object 110.

[0036] A processor 150 may be arranged in data communication with the machine 130 and the platform 140. The processor 150 may be configured to obtain a set of scan parameters as input and provide a set of control data as output, the set of control data comprising at least one of a first control data to control the machine 130 and a second control data to control the platform 140. The processor 150 may be a server computer or part thereof. The server computer may be a single server or comprise a plurality of distributed servers. In some embodiments, the server computer may be a remote server, such as a cloud server.

[0037] In some embodiments, the first control data may comprise a calculated 3D scanner position data (camera pose data) to move the robotic arm to achieve a desired position to scan the object 110. The second control data may comprise an actuator data for rotating the platform 140 about its centre 140 A. The second control data may be sent to a controller of the platform 140, which in turn drives a motor of the platform 140 to rotate the platform 140.

[0038] FIG. 2 A illustrates an embodiment of a method 200 of operating the apparatus 100 in scanning an object 110. The method 200 may be executed with a user interface (UI) for a user to input a set of scan parameters. In some embodiments, the scan parameters may be obtained via a scan parameter estimation algorithm, which may be executed before the 3D scanning. The estimation algorithm may be artificial intelligence based, and may include machine learning, to estimate various dimensions associated with the object.

[0039] In step 202, a user inputs the scan parameters. The scan parameters may include object dimensions comprising a width Wobject, a length lobject, and a height hobject, a desired number of scans (nScans) parameter associated with the 3D scanner 120, and a number of turns parameter (nTurns) associated with the platform 140, into the UI. The user may then indicate that he / she wishes to start scanning by actuating a UI based virtual button, or a physical actuator. If the object is an irregularly shaped object, the width Wobject, the length lobject, and the height hobject parameters may correspond to the maximum Wobject, lobject, and hobject, which may correspond to different points, parts of the object as illustrated in FIG. 2B. In some embodiments, the maximum Wobject, lobject, and hobject may change after rotation of the object110 on the platform 140. In some embodiments, the user may be prompted to input a new set of scan parameters at every rotation of the platform 140 corresponding to a different view of the object 110. In some embodiments, a rotation of the platform 140 may cause the estimation algorithm to output an updated set of scan parameters.

[0040] In step 204, upon a positive determination that the scan parameters are valid, the scan parameters are sent to the machine operating system, such as a robot operating system (ROS) of the robotic arm for the derivation or calculation of one or more operation scanning parameters. In a negative determination, for example, if any of the scan parameters are determined to be invalid, e.g. out of practical range, the user may be prompted to re-start the scanning process with one or more error notifications shown.

[0041] In some embodiments, the operation scanning parameters may include a turning angle, denoted as TurnAngie, and a scanning angle, denoted as ScanAngie. The calculation of TurnAngie and the ScanAngie may be made with reference to FIG. 3 and FIG. 4.

[0042] In step 206, a first position of the 3D scanner is generated and the robotic arm is moved to the first position. An actuating signal / data may be sent to the 3D scanner from the processor 150 to commence scanning.

[0043] In step 208, when the 3D scanning is done at the first position, the robot is moved to the next position based on the calculated ScanAngie, and this continues on until all the nScans positions are scanned for the first position of the object on the turntable.

[0044] In step 210, as the robot is on a single plane, the object 110 has to be turned by the automatic turntable in order to capture all the sides of the object 110. Thus, the next step is the turntable turning a certain degree based on the TurnAngie and the previous camera positions defined in accordance with the ScanAngie are repeated. This may be continued until the object is turned 360 degrees and is completely scanned.

[0045] In step 212, the scanning for the object is completed and a point cloud of the object may be generated.

[0046] In some embodiments, the data may be stored and analysed. A feature extraction algorithm may then be used to extract salient features based on the stored data. Such salient features may include a shape of the object 110, a dimension of the object 110, a material of the object 110 etc.

[0047] FIG. 3 shows an embodiment of the calculation of a maximum angle parameter by the processor 150. The maximum angle parameter may then be used to generate the ScanAngie.The ScanAngie is in turn used to determine the position (camera pose) of the 3D scanner and to actuate the robotic arm accordingly.

[0048] The embodiment may be regarded as an algorithmic generation of camera poses. When the object dimensions are input, the maximum angle (maxAngle) that the robotic arm is able to take to scan may be calculated with reference to FIG 3, which shows the various dimensions and the turntable positioned at a predetermined distance from the robotic arm.

[0049] The circle 302 represents a maximum reach of the robotic arm and a distance factor 304 representative of or related to the optimum focal range of the 3D scanner is subtracted from the distance D between the robot base to the turntable. The subtracted value is then used with a circle equation to get the highest point that the robot can reach for an object of a certain size. From this, the maximum angle for scanning is calculated using trigonometry.

[0050] After calculating or deriving the maximum scanning angle, the scanning angle between each position is calculated by dividing over the number of scans nScans as desired by the user mathematically expressed in Equation (1) as follows:

[0051] The turning angle for the turntable may also be calculated by dividing 360 which is one full revolution, by the number of turns nTurns, mathematically expressed in Equation (2) as follows:TurnAngle =360(2) nTurns

[0052] With the scanning angle previously calculated, the generation of each camera pose for the current scanning stage can be done by as shown by FIG. 4.

[0053] The scan angle may be multiplied by a current stage number of scanning to get the current ScanAngie and the robot x and y coordinate is calculated using trigonometry with a predefined offset distance as the hypotenuse from the centre of the turntable. This offset is directly related to the optimum focal distance of the camera as the calculated pose will be maintained at the offset distance from the centre of the turntable. The resulting robot pose thus uses the x and y coordinates as x and z with y set to the y value of the turntable and the orientation is directly obtained from the current scanning angle.

[0054] In general, the processor 150 is configured to calculate a maximum angle parameter, a scan angle parameter, and a turning angle parameter, wherein the scan angle parameter is calculated based on dividing the maximum angle parameter by the number of scans, and wherein the turning angle parameter is calculated based on dividing 360 by the number of turns parameter. The processor 150 is then configured to calculate a 3D scanner pose parameter based on the scan angle parameter and a predetermined offset distance. The pose parameter is defined by an x-coordinate and a y-coordinate, the x-coordinate and the y-coordinate are calculated based on using the predetermined offset distance as a hypotenuse from the centre 140A.

[0055] In some embodiments, the x-coordinate and y-coordinate of the pose parameter may mathematically expressed in Equation (3) as follows:

[0056] FIG. 5 shows a flowchart of an embodiment of a communication process 500 between the robot arm and 3D scanner. The communication process may be between the ROS and 3D scanner on a user interface (UI) software application via messaging protocol such as MQTT. The communication process may be used to transmit data via wired or wireless communication protocol.

[0057] In step 502, upon receiving a command or instructions from a user to scan the object 110, the start command, scan parameters and whether auto / manual scan data are sent to the robot operating system, which may be a Linux based operation system.

[0058] In step 504, the robotic arm moves to the first position. A controller of the machine may then send a 3D scanner start command (camera start command) and a transformation matrix [1] to the 3D scanner.

[0059] In step 506, the 3D scanner will scan the object 110 and the saves the scan data as a point cloud. Other associated data such as the image data and the received transformation matrix may be saved.

[0060] In step 508, the robotic arm and the turntable will move to the next position of the scanning process, depending on the nScans parameter and the nTurns parameter.

[0061] The steps 506 and 508 will continue until the last scan position.

[0062] In step 510, upon positive determination that the 3D scanning process is at a last position, the start camera, start command and transformation matrix is sent to the UI software application with a termination data indicating that the scanning is at the last position. The data may then be stored, and control may be reverted to the user.

[0063] It is appreciable that during the scanning process, the transformation matrixes that contain the transformation between the camera to the robot base may be saved. The transformation matrix may also contain the rotation of the object from rotating the turntable. This information allows for global alignment of the multiple scans, and after which is closely aligned using Iterative Closest Point (ICP) [1],

[0064] According to another aspect of the disclosure there is a method for generating a 3D scan of an object 110 in an automated disassembly process. FIG. 6 is a flow chart according to another aspect of the disclosure in the form of a computer-assisted method 600 for generating a 3D scan 160 of an object 110 in an automated disassembly process, the method 600 comprising the following steps:

[0065] Step 602: obtaining, by a processor, a set of scan parameters as input;

[0066] Step 604: providing, by the processor, a set of control data as output, the set of control data comprising at least one of a first control data to control a 3D scanner 120 and a second control data to control a machine 130; wherein the machine 130 is configured to move in at least one rotational motion and at least one translational displacement; and the platform 140 is positioned at a pre-determined distance from the machine 130, the platform 140 rotatable about a centre 140 A.

[0067] While the disclosure has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.References[1] P. J. Besl and N. D. McKay, "A method for registration of 3-D shapes", IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, pp. 239-256, Feb. 1992.

Claims

CLAIMS1. An apparatus (100) for generating a 3D scan (160) of an object (110) in an automated disassembly process, the apparatus (100) comprising a 3D scanner (120), the 3D scanner (120) configured to mounted on a machine (130), the machine (130) configured to move in at least one rotational motion and at least one translational displacement; a platform (140) positioned at a distance from the machine (130), the platform (140) rotatable about a centre (140 A); and a processor (150) arranged in data communication with the machine (130) and the platform (140), the processor (150) configured to obtain a set of scan parameters as input and provide a set of control data as output, the set of control data comprising at least one of a first control data to control the machine (130) to move the machine (130) to a desired pose of the 3D scanner, and a second control data to control the platform (140) to move the platform (140) to expose a surface of the object (110) within optical alignment of the 3D scanner.

2. The apparatus (100) of claim 1, wherein the machine (130) is a robotic arm with at least six degree of freedom.

3. The apparatus (100) of claim 1 or 2, wherein the platform (140) comprises a turntable, and turntable configured to support the object (110) and wherein the platform (140) is configured to turn the turntable to expose a part of the object (110) to the 3D scanner (120) for the 3D scan.

4. The apparatus (100) of any one of the preceding claims, wherein the set of scan parameters include object dimensions, a number of scans parameter associated with the 3D scanner (120), and a number of turns parameter associated with the turntable.

5. The apparatus (100) of claim 4, wherein the processor is configured to calculate a maximum angle parameter, a scan angle parameter, and a turning angle parameter, wherein the scan angle parameter is calculated based on dividing the maximum angle parameter by thenumber of scans, and wherein the turning angle parameter is calculated based on dividing 360 by the number of turns parameter.

6. The apparatus (100) of claim 5, wherein the processor (150) is configured to calculate a 3D scanner pose parameter based on the scan angle parameter and a predetermined offset distance.

7. The apparatus (100) of claim 6, wherein the pose parameter is defined by an x- coordinate and a y-coordinate, the x-coordinate and the y-coordinate are calculated based on using the predetermined offset distance as a hypotenuse from the centre (140A).

8. A computer-aided method (600) for generating a 3D scan (160) of an object (110) in an automated disassembly process, the method (600) comprising the steps of: obtaining (602), by a processor, a set of scan parameters as input; providing (604), by the processor, a set of control data as output, the set of control data comprising at least one of a first control data to control a 3D scanner (120) and a second control data to control a machine (130); wherein the machine (130) is configured to move in at least one rotational motion and at least one translational displacement; and the platform (140) is positioned at a pre-determined distance from the machine (130), the platform (140) rotatable about a centre (140A).

9. The method (600) of claim 8, wherein the set of scan parameters include an object dimensions, a number of scans parameter associated with the 3D scanner (120), and a number of turns parameter associated with the turntable.

10. The method (600) of claim 9, further comprising the step of calculating a maximum angle parameter, a scan angle parameter, and a turning angle parameter; wherein the scan angle parameter is calculated based on dividing the maximum angle parameter by the number of scans, and wherein the turning angle parameter is calculated based on dividing 360 by the number of turns parameter.

11. The method (600) of claim 10, further comprising the step of calculating a 3D scanner pose parameter based on the scan angle parameter and a predetermined offset.

12. The method (600) of claim 11, wherein the pose parameter is defined by an x-coordinate and a y-coordinate, the x-coordinate and the y-coordinate are calculated based on using the predetermined offset distance as a hypotenuse from the centre (140A).

13. A computer program, the computer program comprising instructions to execute the computer-assisted method according to any one of claims 8 to 12.

Images