Apparatus, system, and method for generating images of degradation of one or more components of an object

Abstract

Aspects concern an apparatus (100) for generating simulated defect image data (160) of a component (140) of an object (110) for an automated disassembly process, the apparatus (100) comprising a processor (150), the processor (150) configured to: obtain a three-dimensional (3D) image data (202) of the object (110) and a set of 3D coordinates of the component (140) of the object (110); determine, using a component localization algorithm, a set of component data based on the 3D image data (202) and the set of 3D coordinates of the component (140); filter non-component image data from the 3D image data (202), based on the set of component data as input to a feature extraction algorithm, to obtain filtered 3D image data; and render simulated defect image data on the filtered 3D image data to generate the simulated defect image data (160).

Description

APPARATUS, SYSTEM, AND METHOD FOR GENERATING IMAGES OF DEGRADATION OF ONE OR MORE COMPONENTS OF AN OBJECTTECHNICAL FIELD

[0001] This disclosure relates to an apparatus, a system and a method for generating images associated with degradation of a component of an object. The disclosure is particularly suited, but not limited to, generating images simulating one or more stages of degradation of a component of an electronic device to facilitate disassembly of the electronic device.BACKGROUND

[0002] Technological advancement and rising consumer demand have led to increase in the production of electronic devices and other goods, which in turn exponentially increased the volume of end-of-life products or electronic waste (e-waste). Such an increase in e-waste may adversely impact the environment.

[0003] Disassembly processes are typically used to recover valuable components and materials so as to reduce the environmental impact of e-waste and promoting sustainable manufacturing practices. Automated disassembly processes involving one or more robots has emerged as a relatively more efficient solution, but complex product designs, varying levels of product wear and tear, and the requirement for sophisticated sensing and decision-making capabilities may limit the usefulness and / or economic viability of using robots in the disassembly process.

[0004] To complement the robots, computer vision and image processing techniques may be used to aid a robot disassembly system in the identification of components of a waste object to be disassembled. However, the quality and accuracy of such computer vision and image processing techniques may be susceptible to environment lighting and / or uncontrolled reflection off metallic components. In addition, current computer vision and image processing techniques may not adequately take into account degradation associated with one or more components such as degree of wear-and-tear, rust, corrosion, etc.

[0005] There exists a need to provide an improved solution for generating images of degradation of a component, particularly in an automated disassembly process.SUMMARY

[0006] This disclosure was conceptualized to provide a cost-effective image capturing and manipulation solution to simulate wear and tear effects on surfaces of target objects, such as, but not limited to, electric motors, in an automated disassembly process. Such wear and tear effects may have an impact on the automation of the disassembly process. For example, special equipment / considerations may be required to remove one or more defected bolts of an electric motor that are severely corroded by rust or scratches.

[0007] It is desirable to have a solution that automates the process of image capture, such as two-dimensional (2D) image capturing, or three-dimensional (3D) manipulation of a 3D computer-aided design (CAD) images and lighting conditions. In some embodiments, the manipulation of images may include automation of bounding box labeling for one or more components of the target object(s). Such automated bounding box labeling may be achieved using an artificial intelligence (Al) based module. In some embodiments, relatively more realistic looking defects may be generated using a three-dimensional (3D) rendering method / algorithm in a cost-effective manner.

[0008] In one aspect, there is provided an image-based disassembly solution which exploits the use of 3D CAD data with image rendering techniques and search algorithm to enable synthetic defect generation. Notably, the disclosure seeks to improve the robustness and accuracy of an object detection model, used for image-based disassembly of electric motors, in the presence of limited defect data.

[0009] According to an aspect of the present disclosure, an apparatus as claimed in claim 1 is provided. According to another aspect of the present disclosure, a system as claimed in claim 9 is provided. According to another aspect of the present disclosure, a computer-assisted method according to the disclosure is defined in claim 13. A computer program comprising instructions to execute the computer-assisted method is defined in claim 15.

[0010] The dependent claims define some examples associated with the apparatus, system, and method, respectively.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The disclosure will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which:- FIG. 1 is a schematic diagram of an embodiment of an apparatus for generating a simulated defect image of a component of an object for an automated disassembly process;- FIG. 2 is a schematic diagram of an embodiment of a system for generating a simulated defect image of a component of an object for an automated disassembly process;- FIG. 3 is an illustration of an embodiment of a data processing pipeline for generating a simulated defect image of a component of an object;- FIG. 4 is an example of a component identification algorithm, in the form of a k-d search algorithm, with corresponding points in two-dimensional space;- FIG. 5 is an example of the rust defects on a component in the form of an automotive bolt;- FIG. 6 illustrates the localization of a random scratch defect on the exterior of an object, in the form of an electric motor;- FIG. 7 illustrates an example of a graphical user interface, in the form of a UI defects widget;- FIG. 8 illustrates a ray casting concept;- FIG. 9 shows a component localization and segmentation pipeline associated with the ray casting concept; and- FIG. 10 is a flow chart of a method for generating a simulated defect image of one or more components of an object, configured from the UI defects widget, for an automated disassembly process.DETAILED DESCRIPTION

[0012] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. Other embodiments may be utilized and structural, and logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0013] Embodiments described in the context of one of the systems or methods are analogously valid for the other systems or methods.

[0014] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0015] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0016] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0017] As used herein, the term “data” may be understood to include information in any suitable analog or digital form, for example, provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. The term data, however, is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.

[0018] As used herein, the term “module” refers to, or forms part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may include memory (shared, dedicated, or group) that stores code executed by the processor.

[0019] As used herein, the terms ‘first’, ‘second’, ‘third’, and so on, are used for purposes of clarity and do not imply order or precedence.

[0020] As used herein, the term “disassembly process” refers to a process whereby an object is separated into its components. In some embodiments, the disassembly process may be used to separate one or more components (e.g. bolts or other fasteners) from a product (e.g. electric motor) and / or subassemblies via non-destructive or semi-destructive processes which target the connectors / fasteners. Such disassembly process may have applications in various industries, in particular, but not limited to, a waste treatment facility, reverse engineering processes, etc. The disassembly process may broadly include a guided disassembly process using augmented reality, virtual reality, and / or mixed reality; an automated disassembly (i.e. full automation).

[0021] As used herein, the term “three-dimensional (3D) scanner” may refer to one or more image capturing devices, and may include various 3D scanning techniques. In some embodiments, the 3D scanners include structured light 3D scanners, which can be configured to collect high quality data. In some embodiments, the 3D scanning techniques may be based on stereo vision methods, and / or time-of-flight methods.

[0022] It is envisaged that structured light 3D scanners, may produce scanning having relatively higher precision and / or accuracy. A structured light 3D scanner may be configured to work using principles of triangulation. A light source of the 3D scanner may project a fringe pattern across the scan target object surface, and two cameras of the 3D scanner may be used to capture the surface geometry based on the pattern distortion, calculating 3D coordinate measurements. The 3D scanner may process coordinate data into a "point cloud," creating a digital image of the object. In some embodiments, the light source of the 3D scanners may be configured to emit white light. In some embodiments, the light source may be configured to emit other light such as blue light, which may be used to capture data on shinier and darker colored surfaces and filters out the ambient light.

[0023] As used herein, the term “processor” refers to a circuit, including analog circuits or components, digital circuits or components, or hybrid circuits or components. Any other kind of implementation of the respective functions which will be described in more detail below may also be understood as a "circuit" in accordance with an alternative embodiment. A digital circuit may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, or a firmware.

[0024] As used herein, the term “obtain”, as used herein, refers to the processor which actively obtains the inputs, or passively receives inputs from a user interface and / or one or more sensors. The term obtain may also refer to the processor which receives or obtains inputs from a communication interface, e.g. the user interface. The processor or the dosing module may also receive or obtain the inputs via a memory, a register, and / or an analog-to-digital port.

[0025] An embodiment of the disclosure is shown in FIG. 1, which illustrates a setup of an apparatus 100 for generating a simulated defect image 160 of a component 140 of an object 110. The apparatus 100 may be suited for an automated disassembly process. In some embodiments, the automated disassembly process may be utilized for the disassembly of an electric motor, and may include the removal of one or more components 140, such as one ormore bolts, from the electric motor. In some embodiments, the automated disassembly process may form part of a waste or materials recovery facility, such as an e-waste recovery facility.

[0026] The apparatus 100 may comprise a processor 150, the processor 150 configured to: obtain one or more three-dimensional (3D) image data 102 of the object 110 and a set of 3D coordinates 104 of the component 140 of the object 110; determine, using a component localization algorithm, a set of component data based on the 3D image data 102 and the set of 3D coordinates of the component 140; filter non-component image data from the 3D image data 102, based on the set of component data using a feature extraction algorithm (for example, fed as input to the feature extraction algorithm), to obtain filtered 3D image data; and render simulated defect image data on the filtered 3D image data to generate the simulated defect image 160. The filtered 3D image data may be an extracted component image data. In some embodiments, the set of component data may include the Cartesian coordinates of one or more vertex of the component 140. In some embodiments, the feature extraction algorithm may be executed by another processor, i.e. the set of component data may be sent to the another processor or the another processor may be configured to obtain the set of component data.

[0027] In some embodiments, the apparatus 100 further comprises a graphical user interface (GUI) 170 arranged in data communication with the processor 150, the GUI 170 configured to display a set of image parameters associated with the simulated defect image data for input by a user. The set of parameters may include a first image parameter 172 associated with a degree of defect, and a second image parameter 174 associated with an environment. The degree of defect may include examples such as rust, scratches, or other forms of corrosion such as screw / bolt head damage, etc. The parameter associated with the environment may be a degree of lighting or light intensity surrounding the detect image data. The set of parameters may be used by a user to customize the degree of defect, for example, a percentage (%) of rust on a component, one or more defacement (e.g. scratch marks, scuff marks) of the component, and the light intensity akin to brightness and / or contrast adjustment of the simulated defect image.

[0028] In some embodiments, the set of component data is obtained based on a comparison data between the set of 3D coordinates of the component 140 and the data of a trained component database.

[0029] In some embodiments, the component localization algorithm comprises a spacepartitioning data structure, the space-partitioning data structure configured to receive the set of3D coordinates of the component (140) as input to output the set of component data. The spacepartitioning data structure may be generated by a k-dimensional tree (KDTree) algorithm.

[0030] In some embodiments, the processor is configured to render the simulated defect image data using a computer graphics rendering algorithm, such as a ray casting algorithm.

[0031] FIG. 2 shows a system 200 for generating a simulated defect image of a component 140 of an object 110 for an automated disassembly process. The system 200 may include an image capturing device 202, and may incorporate the apparatus 100 to process images captured by the image capturing device 202. The system 200 may be suited for the automated disassembly of an electric motor 204 using a robotic solution.

[0032] The image capturing device 202 may include a three-dimensional (3D) scanner and an RGB image capturing device (e.g. a camera). The 3D scanner with a two-in-one scanner device, comprising of a 3D scanning device for point cloud data collection and an RGB camera to capture 2D images. In the context of an automated disassembly of the electric motor 204, the components to be disassembled from the electric motor 204 may include one or more bolts 206.

[0033] In some embodiments, the 3D scanner may be mounted on a robotic arm 208. The robotic arm 208 may be configured to move in at least one rotational motion and at least one translational displacement. One or more platforms (not shown) may be made available for the object 110 to be positioned thereon so that 3D scans may be obtained.

[0034] In some embodiments, the 3D scanner may be a structured light 3D scanner.

[0035] In some embodiments, the robotic arm 208 may be a robotic arm with six degree of freedom. In some embodiments, the robotic arm may be a six-axis articulated robotic arm, and may comprise one or more controllers for communication with the processor 150. It is appreciable that the robotic arm may have higher degrees of freedom, i.e. more than 6 degrees of freedom.

[0036] In some embodiments, the robotic arm 208 may be a collaborative robot (cobot), the cobot being intended for direct human-robot interaction within a shared space or shared area. Such cobot may be especially suited in situations where human users / controllers and the robots are in close proximity. The cobot may be formed from lightweight construction materials, rounded edges, and may comprise a limitation of speed and force to enhance safety during operation.

[0037] In some embodiments, the robotic arm may comprise sensors (e.g. laser displacement sensors) and software that enhance safe behaviour during usage. For example, the robotic arm 208 may be configured to stop operation when it detects an obstacle (e.g. another robot, human user) in its movement path. In some embodiments, the robotic arm may be equipped with additional components such as force sensors, torque sensors, tool changer, a customized mounting frame for mounting the 3D scanner and other robot locomotion, such as unbolting components / sensors, diagnosis components, etc.

[0038] In some embodiments, the robotic arm 208 may be equipped for the disassembly of the electric motor components, which include, but are not limited to, the removal of coil windings, differential gears, and unscrewing of exterior automotive bolts. It may be appreciable that the simplicity and homogeneity of the electric motor’s internal components may indicate a simpler way to remove the internal components than the exterior encasement, which is frequently exposed to unknown environmental conditions.

[0039] The system 200 depicted in FIG. 2 may be configured to receive 3D image data 102 of the electric motor 204 and coordinates information 104 relating to the one or more bolts 206. Based on the image data of the object 110, the set of component data relating to the one or more components 140 may be derived or generated, and the relatively realistic looking (i.e., synthetic) deterioration of the components, for example bolts, may be overlaid on the components 140. In some embodiments, one or more 2D images of the object 110 may be captured under different light conditions, and the components may be manually labelled or annotated for subsequent training of the component localization algorithm and / or the feature extraction algorithm.

[0040] As depicted in FIG. 2, there comprises an edge computing device 150A, the edge computing device 150A comprises a robot control module configured to send control signals to control the robotic arm; and a component detection module, the component detection module configured to determine, using the component localization algorithm, a bounding box around each of the one or more bolts on the electric motor, and the set of 3D coordinates of each of the one or more bolts.

[0041] The system 200 further comprises an edge computing server 150B, the edge computing server 150B comprises an Al-based detection module. The apparatus 100 may be used to train the edge server 150B, before deployment on an edge computer 150A, on a periodic or need-to-basis. It may be appreciated that system 200 may integrate the apparatus 100 into apractical disassembly system solution context, wherein the simulated defect images 160 generated can be used to train a vision-based bolt detection Al algorithm, deployed on the edge server 150B, before deployment on an edge computer 150A, on a periodic or as need basis. In some embodiments, the Al based module may be trained to perform bounding box labelling using a pretrained Al model’s predictions as labels. These labels may be prior reviewed and corrected by human and use to train the Al model. In some embodiments, the labelling may be automatically performed, i.e. directly use the model’s predictions as labels, no review or correction needed. In some embodiments, crowd-sourcing may be used, i.e. distributing the data labelling task to a large group of people via online platforms.

[0042] FIG. 3 depicts a data processing flow diagram 300, in the form of stages, for the automatic synthetic defect generation solution. Each stage may be in the respective subsections. For ease of understanding, the object 110 is an electric motor, and the described component 140 of the object 110 may include one or more automotive bolts, which may be common in modern vehicles.

[0043] In stage 1, 3D image data 102, in the form of a 3D computer-aided design (CAD) data 302 is obtained. In relation to 3D coordinates 304 of the components, and considering the feature complexity of the automotive bolts against the electric motor in 3D space, the component’s centre coordinates (i.e., x coordinate, y coordinate, z coordinate) may be required. The 3D coordinates may be obtained from known sources such as a 3D CAD model component list / technical specifications / operating manuals of the electric motor. In some embodiments, a 3D CAD model of an object (e.g. electric motor) may be loaded into a compatible CAD software. Within the CAD software, a selection tool may be used to pinpoint an automotive bolt on the electric motor, and the component identifier may be shown and obtained. For instance, a selection tool may pinpoint a bolt having m5bolt_id_X, where the X suffix denotes the bolt’s ID. Modem CAD tools allow users to filter out unrelated 3D components based on the specified prefixes and / or suffixes. A typical automotive bolt often has a cylindrical or hexagonal like surface. Once the bolt is selected, the corresponding 3D coordinates can be retrieved from a properties or information panel within the CAD software. Assuming the bolt’s top has a circular shape, its 3D representation aligns mathematically with the cylinder’s equation, x2+2= r2, where r is the radius. The z-axis pinpoints the bolt’s unique location relative to the model’s defined origin point. For modem CAD software, these cartesian coordinates can be easily determined via the measure or evaluate tool feature. Theaforementioned methodology may be applied to the remaining components (e.g. other bolts) to retrieve their coordinates. Notably, some CAD software offer automated listing or export functions, further streamlining the process and may save time.

[0044] Thereafter, the 3D CAD data 302 and the 3D coordinates 304 of the component 140 may be output to Stage 2 for further processing.

[0045] Stage 2 comprises performing pairwise matching of 3D CAD data 302 with the bolt coordinate information 304. Due to the complex bolt geometry in 3D space, a component localization algorithm 306 may be configured to perform fine-grained localization for either single or multiple bolts. Additional image fine-tuning / capturing may be effected to capture relevant 3D bolt surfaces and contours, before mapping to individual identifiers of each component, for example, a unique ID associated with each bolt.

[0046] Stage 3 may include three modules: an automatic feature extraction module 308, a bounding box annotation module 310, and a 3D defect render parameters module 312. Based on the output of Stage 2 and the angle of the image capturing device 202, the received component information, e.g. cartesian coordinate information of the component (e.g. bolt cartesian coordinate information) or target geometry data in Fig 9, may be used as input to the automatic feature extraction module 308, the automatic feature extraction module 308 used to automatically filter away non-essential 3D surfaces, leaving only the bolts. Note that while the image capturing device 202 operates in 3D space coordinates, the output may be a 2D image- this may be akin to how a camera works in reality.

[0047] The bounding box annotation module 310 may be configured to receive a-priori information of the component 140, such as the bolt size. The bounding box annotation module 310 may comprise mathematical operators, such as minmax mathematical operators and simple trigonometry, applied to the 3D coordinates 304, to obtain bounding box coordinates of each component 140 in an automated manner.

[0048] The 3D defect render parameters module 312 may be configured to effect corresponding defect types and intensity on the bolt surface. The defect type and intensity may be based on the first image parameter and the second image parameter. Examples of rendered defects are rust, scratch, and colour. In some embodiments, at least one of the first parameter and the second parameter may be normalized to a range of 0 to 1.

[0049] In stage 4, the output from the modules 308, 310, 312 may be input into a 3D rendering engine 314 and UI defects widget 316. The 3D rendering engine 314 may beconfigured to overlay the visual effects, received from Stage 3, onto one or more components 140. To inspect the realism of the defects rendered, a user can launch the UI defects widget 316 via a key combination, such as ‘Ctrl + Shift + F’. In some embodiments, the key combination is user-configurable. The defect type and intensity may be changed by the user with the updated changes reflected in real-time. Likewise, defects for individual bolts can also be customized using this UI defect widget.

[0050] In stage 5, the user may be required to determine the number of pairwise pictures and labels / annotations to generate as an input to a generator application program. For example, a pre-determined number of samples, such as 500 or 5000 samples of images with varying ambient parameters, such as rust, scratch, and illumination intensity may be generated and prestored in a database. Thereafter, the pairwise data are output to a folder, wherein the user can retrieve the image and import as a training dataset for the desired object detection model.

[0051] In some embodiments, additional parameters may be used for tweaking or finetuning. Such additional parameters include, but are not limited to, camera angles and the external light source illumination angles, to simulate changing lighting conditions in practice, which may impact the accuracy of any object detection model.

[0052] The output degenerated from stage 5 comprises an output set 318 comprising 2D simulated defect images 160 of the bolts 206 and bounding box labels around each bolt 206.

[0053] In the various described embodiments, the component localization algorithm may include a space-partitioning data structure, the space-partitioning data structure configured to receive the set of 3D coordinates of the component 140 as input to output the set of component data. The space-partitioning data structure may be generated using a k-dimensional tree (KDTree) algorithm [1] as illustrated in FIG. 4 [2][3] . In some embodiments, the k-dimensional tree (KDTree) algorithm may localize components and graphically render specific defects using real-world textures. In particular, the use of ray casting algorithm [4] within a 3D computer graphics software Blender [5] used for creating visual effects, animated films, video games, may be used.

[0054] It is appreciable that the data labelling / annotation of the targeted objects may be automated with relatively higher accuracy whilst improving the human productivity levels for highly repetitive task of data annotation.

[0055] In some embodiments, the KDTree algorithm may be configured as a data structure that effectively represents nodes in a multidimensional space. It is used extensively in computergraphics, machine learning, and computational geometry for tasks such as nearest neighbour search and range search. The k-d tree represents points as nodes in a binary tree.

[0056] By partitioning the elements along various dimensions of space, the tree may be constructed recursively. At each level, the dividing dimension varies, resulting in a balanced tree, see FIG. 4 for illustration.

[0057] To construct the tree, a set of points may be selected and a dimension chosen / selected for dividing them. As shown in FIG. 4, based on the median value along that dimension, the elements may be divided into two subsets. This procedure may be repeated recursively until a termination criterion is met for each subset, and facilitates efficient search point queries upon completion of the tree-like structure [3], A nearest neighbour search can then be used for traversing the tree structure by comparing points along the dividing dimensions and selectively investigating pertinent subsets to restrict the search space. The closest point discovered is frequently maintained and updated whenever a closer point is identified.

[0058] In some embodiments, a range search may be performed by selectively investigating tree subsets that intersect with the specified range, thereby efficiently identifying points within the range. In this case, it may be inferred that no two points share the same 3D coordinates (i.e., x, y, z). Consequently, the k-d tree algorithm may be regarded as an effective method for organizing points in multidimensional space, facilitating quick and accurate querying. In particular, KDTree’s recursive nature and partitioning strategy makes it applicable to a variety of research domains, thereby enhancing performance and facilitating advanced data analysis.

[0059] In the electric motor disassembly context, the KDTree method may be applied using the centre 3D coordinates for each bolt of interest and apply for visually observable adjacent surfaces (or pertinent vertex points). The mapping of KDTree outputs to individual bolts is obtained.

[0060] In some embodiments, the defects rendering algorithm may include the Blender software. The Blender software offers a variety of tools and techniques to create defects in rendered images, such as texture maps, procedural shaders, or post-processing effects. Defects rendering in Blender is the intentional addition of imperfections to create a more vibrant and lifelike virtual environment. By simulating defects or deterioration such as scratches, dirt, and other natural artifacts, a user may add depth and authenticity into their digital creations.

[0061] In some embodiments, tools like texture maps and procedural shaders can enhance the appearance of objects and materials. FIG. 5 illustrates the different intensity of rustrendering on all surfaces of the automotive bolt, which may be simulated using both the fractal Perlini noise function [5], Based on the output of the KDTree, various degrees of bolt defect is now applicable either uniformly across one or more bolts.

[0062] Furthermore, the user can choose to randomly apply defects, such as scratch, on several bolts at once, as shown in FIG. 6.

[0063] In some embodiments, the UI defects widget 316 may be implemented as a slider widget for dynamic visualization of wear-and-tear effects as shown in FIG. 7.

[0064] The proposed slider-based user interface (UI) shown in FIG. 7 may offer an intuitive, seamless experience for users regardless of their technical expertise. The UI may presents data in a relatively easy-to-understand manner, enabling rapid simulation of complex real-world conditions and extraction of the desired defect parameter ranges.

[0065] In some embodiments, the users may simply input the required defect parameters, individual parameter range (where applicable), and number of images into the UI defects widget 316 to generate the 2D image results and to streamline the entire process.

[0066] In some embodiments, the UI defects widget may be programmed using a high-level programming language, such as the python programming language.

[0067] In some embodiments, the bounding box annotation module 310 may be configured to annotate the components, such as partially occluded components, based on the ray casting algorithm [4], The ray casting algorithm is a tool for 3D image rendering before ray tracing immerse with the computational power of Graphics Processing Units. Referring to FIG. 8, for each pixel of the rendered image 802, a ray may be cast onto the objects, and calculate the intersection 804 of the ray to each object’s face (i.e., surface). Given the camera intrinsics, the direction of the ray casted through pixel (x, y) in the rendered image may be calculated, and the direction vector of the ray may be mathematically expressed as Equation (1) as follows:where i , j, dcam are camera intrinsic vectors corresponding to focal length, aperture, field-of- view, resolution, and the origin of the ray (s) is camera position scam. Thus, the ray can be represented as a parametric equation+dcamwhere d is the direction vector of the ray in Equation (1) and scamis the camera position 806, see FIG. 8. Each face (surface) of the objects, that intersects with the casted ray of the camera 808 can be represented as an implicit equation nP + D = owhere n is the object face’s 3D normal vector, P is the 3D vectorcoordinate to a point on the face, and D is the distance from the face to the origin. Solving the equation system enables the intersection to be found, and verify whether the intersection is in the face and t in [t min, t max], It may be appreciable that the 3D mesh may be regarded as a combination of faces (plane), and therefore the pixel to an object can be assigned based on the closest intersection.

[0068] As shown in FIG. 9, ray casting 902 may provide or output a segmentation map 904 of the rendered image. Using the geometry data (e.g. 3D coordinates) of the interested targets 906, which is provided from the scanning step, the mask information for each target object, such as a cube or cuboid shown in FIG. 9, can then be extracted. A segmentation mask 908 of the cube or cuboid contains pixel coordinates and can be represented as two one-dimensional (ID) arrays, i.e. [xi, X2, ..., xn] and [yi, y2, ...,yn]. The minimum and maximum values of each ID array may be extracted to infer the bounding box coordinates 910 of the object, see in conjunction FIG. 4, using the KDTree algorithm. In other words, the bounding box coordinates may be derived from the segmentation mask.

[0069] While the ray casting algorithm is simple but computationally expensive, and the complexity of the algorithm increases with the image resolution and number of primitive surfaces in the object space. For example, the CAD model have at least one million surfaces and one million rays (i.e., 1 X 106) may be cast for an image resolution of 1000 pixels by 1000 pixels. The Blender software facilitates efficiency by introducing acceleration structures such as Bounded Volume Hierarchy (BVH) [6], These structures are may be constructed using the 3D geometry of the objects that are visible in the scene. While casting a ray, the structure enables the removal of unnecessary surfaces and saves computing cost.

[0070] According to another aspect and with reference to FIG. 10, there is provided a computer-aided method 600 for generating a simulated defect image of a component 140 of an object 110 for an automated disassembly process, the method 600 comprising the steps of: obtaining 602 a three-dimensional (3D) image data 102 of the object 110 and a set of 3D coordinates of the component 140 of the object 110; determining 604, using a component localization algorithm, a set of component data based on the 3D image data 102 and the set of 3D coordinates of the component 140; filtering 606 non-component image data from the 3D image data 102, based on the set of component data as input to a feature extraction algorithm, to obtain filtered 3D image data; rendering 608 simulated defect image data on the filtered 3D image data; generating 610 the simulated defect image 160 based on the rendered simulateddefect image data. In some embodiments, the simulated defect image 160 comprises a 2D image and a label data. In some embodiments, the method 600 may be implemented as a computer program for installation on the processor 150, the computer program comprising instructions to execute the method 600. The filtered 3D image data may be an extracted component image data.References[1] Bentley, J.L., 1975. Multidimensional binary search trees used for associative searching. Communications of the ACM, 18(9), pp.509-517.[2] Anzola, J., Pascual, J., Tarazona, G. and Gonzalez Crespo, R., 2018. A clustering WSN routing protocol based on kd tree algorithm. Sensors, 18(9), p.2899.[3] Kakde, H.M., 2005. Range searching using kd tree. Florida State University.[4] Scott D Roth, Ray casting for modeling solids, Computer Graphics and Image Processing, Volume 18, Issue 2, 1982, Pages 109-144, ISSN 0146-664X, https: / / doi.org / 10.1016 / Q146- 664X(82)90169-l.[5] Community, B.O., 2018. Blender - a 3D modelling and rendering package, Stichting Blender Foundation, Amsterdam. Available at: https: / / docs.blender.Org / manual / en / 3.6 / render / freestyle / viewJayer / line style / modifiers / geom etry / perlin noise J2d.html[6] Gunther, J., Popov, S., Seidel, H.P. and Slusallek, P., 2007, September. Realtime ray tracing on GPU with BVH-based packet traversal. In 2007 IEEE Symposium on Interactive Ray Tracing (pp. 113-118). IEEE.

Claims

CLAIMS1. An apparatus (100) for generating simulated defect image data (160) of a component (140) of an object (110) for an automated disassembly process, the apparatus (100) comprising a processor (150), the processor (150) configured to: obtain a three-dimensional (3D) image data (102) of the object (110) and a set of 3D coordinates (104) of the component (140) of the object (110); determine, using a component localization algorithm, a set of component data based on the 3D image data (102) and the set of 3D coordinates (104) of the component (140); filter non-component image data from the 3D image data (102), based on the set of component data using a feature extraction algorithm, to obtain filtered 3D image data; and render simulated defect image data on the filtered 3D image data to generate the simulated defect image data (160).

2. The apparatus (100) of claim 1, wherein the simulated defect image (160) comprises a two-dimensional (2D) image data and a label data associated with the component (140).

3. The apparatus (100) of claim 1 or 2, wherein the apparatus comprises a graphical user interface (GUI) (170), the GUI (170) configured to display a set of parameters (172, 174) associated with the simulated defect image data for input by a user.

4. The apparatus (100) of any one of the preceding claims, wherein the set of component data is obtained based on a comparison data between the set of 3D coordinates (104) of the component (140) and the data of a trained component database.

5. The apparatus (100) of any one or the preceding claims, wherein the component localization algorithm comprises a space-partitioning data structure, the space-partitioning data structure configured to receive the set of 3D coordinates of the component (140) as input to output the set of component data.

6. The apparatus (100) of claim 5, wherein the space-partitioning data structure is generated using a k-dimensional tree (KDTree) algorithm.

7. The apparatus (100) of any one of the preceding claims, wherein the processor is configured to render the simulated defect image data using a computer graphics rendering algorithm.

8. The apparatus (100) of claim 7, wherein the computer graphics rendering algorithm is a ray casting algorithm.

9. An automated disassembly system (200), the system (200) comprising the apparatus (100) of any one of the preceding claims, wherein the simulated defect image data (160) output from the apparatus (100) is configured as training input to train a vision-based component detection Al algorithm.

10. The system (200) of claim 9, further comprising a three dimensional scanner and an RGB image capturing device (202), configured to obtain three-dimensional (3D) image data of an object (110) and a set of 3D coordinates (104) of the component of the object (110), wherein the object (110) is an electric motor (204), and the component (140) comprises one or more bolts (206) of the electric motor (204), wherein the system (200) comprises a robotic arm (208) operable to remove the one or more bolts (206), and wherein the three-dimensional scanner and the RGB image capturing device (202) are mounted on the robotic arm (208).

11. The system (200) of claim 10, wherein the system (200) further comprises an edge computing server (150B), the edge computing server (150B) comprises the Al-based detection module.

12. The system (200) of claim 10 or 11, wherein the robotic arm (208) is arranged in data communication with an edge computing device (150A), the edge computing device (150A) comprises a robot control module configured to send control signals to control the robotic arm; and a component detection module, the component detection module configured to determine, using the component localization algorithm, a bounding box around each of the oneor more bolts on the electric motor, and the set of 3D coordinates of each of the one or more bolts.

13. A computer-aided method (600) for generating simulated defect image data of a component (140) of an object (110) for an automated disassembly process, the method (600) comprising the steps of: obtaining (602) a three-dimensional (3D) image data (202) of the object (110) and a set of 3D coordinates of the component (1 0) of the object (110); determining (604), using a component localization algorithm, a set of component data based on the 3D image data (202) and the set of 3D coordinates of the component (140); filtering (606) non-component image data from the 3D image data (202), based on the set of component data as input to a feature extraction algorithm, to obtain filtered 3D image data; rendering (608) the simulated defect image data on the filtered 3D image data; and generating (610) the simulated defect image (160) based on the rendered simulated defect image data, wherein the simulated defect image (160) comprises one or more component label data.

14. The computer-aided method of claim 13, wherein the simulated defect image comprises a two-dimensional (2D) image data and a label data being automatically associated with the respective components (140).

15. A computer program, the computer program comprising instructions to execute the computer-assisted method according to claim 13 or 14.

Images