Apparatus, system, and method for generating degraded images of one or more components of an object

Abstract

Aspects relate to an apparatus (100) for generating simulated defect image data (160) for components (140) of an object (110) for an automated disassembly process, the apparatus (100) comprising a processor (150) configured to: obtain three-dimensional (3D) image data (202) of the object (110) and a set of 3D coordinates of components (140) of the object (110); determine a set of component data based on the 3D image data (202) and the set of 3D coordinates of the components (140) using a component localization algorithm; filter non-component image data from the 3D image data (202) based on the set of component data as input to a feature extraction algorithm to obtain filtered 3D image data; and, render simulated defect image data on the filtered 3D image data to generate simulated defect image data (160).

Description

Technical Field

[0001] This disclosure relates to apparatus, systems, and methods for generating images associated with the degradation of components of an object. This disclosure is particularly applicable to, but not limited to, generating images simulating one or more stages of degradation of components of an electronic device to facilitate the disassembly of the electronic device. Background Technology

[0002] Technological advancements and growing consumer demand have led to increased production of electronic devices and other goods, which in turn has resulted in an exponential increase in the total amount of end-of-life products, or e-waste. This increase in e-waste can have adverse environmental impacts.

[0003] Disassembly processes are often used to recover valuable components and materials in order to reduce the environmental impact of e-waste and promote sustainable manufacturing practices. Automated disassembly processes involving one or more robots have become a relatively more efficient solution; however, complex product designs, varying degrees of product wear and damage, and the requirements for sophisticated sensing and decision-making capabilities may limit the practicality and / or economic feasibility of using robots in disassembly processes.

[0004] As an adjunct to robots, computer vision and image processing technologies can be used to assist robotic dismantling systems in identifying components of waste materials to be dismantled. However, the quality and accuracy of such computer vision and image processing technologies may be susceptible to ambient lighting and / or uncontrolled reflections from metal components. Furthermore, current computer vision and image processing technologies may not adequately account for the degree of degradation associated with one or more components, such as wear and damage, rust, corrosion, etc.

[0005] There is a need for an improved solution, particularly for generating images of component degradation during automated disassembly processes. Summary of the Invention

[0006] This disclosure is conceptualized to provide a cost-effective image capture and manipulation solution for simulating abrasion and damage effects on the surface of a target object, such as, but not limited to, an electric motor, during automated disassembly. Such abrasion and damage may impact the automation of the disassembly process. For example, special equipment / considerations may be required to remove one or more defective bolts from an electric motor that is severely corroded by rust or scratches.

[0007] The desired solution is a solution for automating the process of image capture (such as 2D image capture) or 3D computer-aided design (CAD) images and the manipulation of lighting conditions in three dimensions. In some embodiments, image manipulation may include the automation of bounding box annotation of one or more components of a target object(s). Such automated bounding box annotation can be achieved using artificial intelligence (AI) based modules. In some embodiments, 3D rendering methods / algorithms can be used to generate defects that appear relatively more realistic in a cost-effective manner.

[0008] In one aspect, an image-based disassembly solution is provided, which utilizes 3D CAD data, image rendering techniques, and search algorithms to generate synthetic defects. Notably, this disclosure seeks to improve the robustness and accuracy of target detection models used for image-based disassembly of electric motors, even in the presence of limited defect data.

[0009] According to one aspect of this disclosure, an apparatus as claimed in claim 1 is provided. According to another aspect of this disclosure, a system as claimed in claim 9 is provided. According to another aspect of this disclosure, a computer-aided method according to this disclosure is defined in claim 13. According to claim 15, a computer program comprising instructions for performing the computer-aided method is defined.

[0010] The dependent claims define examples associated with apparatus, systems, and methods, respectively. Attached Figure Description

[0011] This disclosure will be better understood when considered in conjunction with non-limiting examples and the accompanying drawings, in which: Figure 1 This is a schematic diagram of an embodiment of a device for generating simulated defect images of components of an object in an automated disassembly process; Figure 2 This is a schematic diagram of an embodiment of a system for generating simulated defect images of components of an object in an automated disassembly process; Figure 3 This is a diagram of an embodiment of a data processing pipeline for generating simulated defect images of components used to generate objects; Figure 4 This is an example of a component recognition algorithm in the form of a kd search algorithm, with corresponding points in two-dimensional space; Figure 5 This is an example of rust defects on components in the form of automotive bolts; Figure 6 The illustration shows the location of random scratch defects on the exterior of an object in the form of an electric motor; Figure 7 The illustration shows an example of a graphical user interface in the form of UI defect controls; Figure 8 The diagram illustrates the concept of light projection; Figure 9 This illustrates a component positioning and segmentation pipeline associated with the concept of light projection; and Figure 10 This is a flowchart of a method for generating simulated defect images of one or more components of an object configured from a UI defect control for an automated disassembly process. Detailed Implementation

[0012] The following detailed description refers to the accompanying drawings, which illustrate specific details and embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure. Other embodiments may be utilized, and structural and logical changes may be made, without departing from the scope of the present disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.

[0013] The embodiments described in the context of one of the systems or methods are similarly effective for other systems or methods.

[0014] Features described in the context of one embodiment may be applied correspondingly to the same or similar features in other embodiments. Features described in the context of one embodiment may be applied correspondingly to other embodiments, even if not explicitly described in those other embodiments. Furthermore, additions and / or combinations and / or substitutions as described for a feature in the context of one embodiment may be applied correspondingly to the same or similar features in other embodiments.

[0015] In the context of various embodiments, the articles “a,” “an,” and “the” used with respect to a feature or element include references to one or more features or elements.

[0016] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0017] As used herein, the term "data" can be understood to include information in any suitable analog or digital form, provided, for example, as a file, a part of a file, a collection of files, a signal or stream, a part of a signal or stream, a collection of signals or streams, and so on. However, the term data is not limited to the examples mentioned above and can take various forms and represent any information as understood in the art.

[0018] As used herein, the term "module" refers to or comprises, or includes, any of the following: application-specific integrated circuits (ASICs); electronic circuits; combinational logic circuits; field-programmable gate arrays (FPGAs); processors (shared, dedicated, or grouped) that execute code; other suitable hardware components that provide the described functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip. The term "module" may include memory (shared, dedicated, or grouped) that stores code executed by a processor.

[0019] As used herein, the terms “first,” “second,” “third,” etc., are used for clarity and do not imply any order or priority.

[0020] As used herein, the term "disassembly process" refers to the process by which an object is separated into its components. In some embodiments, a disassembly process can be used to separate one or more components (e.g., bolts or other fasteners) from a product (e.g., an electric motor) and / or subassemblies through a non-destructive or semi-destructive process targeting connectors / fasteners. Such disassembly processes can have applications in a variety of industries, particularly but not limited to waste disposal facilities, reverse engineering processes, etc. Disassembly processes can be broadly categorized to include guided disassembly processes using augmented reality, virtual reality, and / or mixed reality; automated disassembly (i.e., fully automated).

[0021] As used herein, the term "3D scanner" may refer to one or more image capture devices and may include a variety of 3D scanning techniques. In some embodiments, the 3D scanner includes a structured light 3D scanner, which may be configured to acquire high-quality data. In some embodiments, the 3D scanning technique may be based on stereo vision methods and / or time-of-flight methods.

[0022] Imagine a structured light 3D scanner capable of producing scans with relatively high precision and / or accuracy. The structured light 3D scanner can be configured to operate using the principles of triangulation. The scanner's light source projects a striped pattern across the surface of the target object, and the scanner's two cameras can be used to capture surface geometry based on pattern distortion, thereby calculating 3D coordinate measurements. The 3D scanner can process the coordinate data into a "point cloud," creating a digital image of the object. In some embodiments, the 3D scanner's light source can be configured to emit white light. In some embodiments, the light source can be configured to emit other light, such as blue light, which can be used to capture data on brighter and darker colored surfaces and filter out ambient light.

[0023] As used herein, the term "processor" refers to a circuit, including analog circuits or components, digital circuits or components, or hybrid circuits or components. According to alternative embodiments, any other kind of implementation of the various functions, which will be described in more detail below, may also be understood as a "circuit". A digital circuit can be understood as any kind of logical implementation entity, which may be a dedicated circuit or a processor or firmware that executes software stored in memory.

[0024] As used herein, the term “acquire” refers to a processor that actively acquires input or passively receives input from a user interface and / or one or more sensors. The term “acquire” may also refer to a processor that receives or acquires input from a communication interface (e.g., a user interface). Processors or metrology modules may also receive or acquire input via memory, registers, and / or analog-to-digital ports.

[0025] exist Figure 1 Embodiments of this disclosure are illustrated, showing the configuration of an apparatus 100 for generating a simulated defect image 160 of component 140 of object 110. The apparatus 100 can be adapted for automated disassembly processes. In some embodiments, the automated disassembly process can be used for disassembling an electric motor and may include removing one or more components 140, such as one or more bolts, from the electric motor. In some embodiments, the automated disassembly process may be part of a waste or material recycling facility, such as an electronic waste recycling facility.

[0026] The apparatus 100 may include a processor 150 configured to: obtain one or more three-dimensional (3D) image data 102 of object 110 and a set of 3D coordinates 104 of component 140 of object 110; determine a set of component data based on the 3D image data 102 and the set of 3D coordinates of component 140 using a component localization algorithm; filter non-component image data from the 3D image data 102 based on the set of component data using a feature extraction algorithm (e.g., fed as input to the feature extraction algorithm) to obtain filtered 3D image data; and render simulated defect image data on the filtered 3D image data to generate a simulated defect image 160. The filtered 3D image data may be the extracted component image data. In some embodiments, the set of component data may include Cartesian coordinates of one or more vertices of component 140. In some embodiments, the feature extraction algorithm may be performed by another processor, i.e., the set of component data may be sent to the other processor, or the other processor may be configured to obtain the set of component data.

[0027] In some embodiments, the apparatus 100 further includes a graphical user interface (GUI) 170 arranged to communicate data with the processor 150. The GUI 170 is configured to display a set of image parameters associated with simulated defect image data for user input. This set of parameters may include a first image parameter 172 associated with the degree of the defect and a second image parameter 174 associated with the environment. The degree of the defect may include examples such as rust, scratches or other forms of corrosion, or damage to screw / bolt heads. The environment-associated parameters may be the intensity of light or illumination around the detected image data. This set of parameters can be used by the user to customize the degree of the defect, for example, the percentage (%) of rust on the component, wear on one or more surfaces of the component (e.g., scratch marks, abrasion marks), and light intensity similar to adjustments for brightness and / or contrast of the simulated defect image.

[0028] In some embodiments, the set of component data is obtained based on comparison data between the set of 3D coordinates of component 140 and data from a trained component database.

[0029] In some embodiments, the component localization algorithm includes a spatial partitioning data structure configured to receive the set of 3D coordinates of the component (140) as input and output the set of component data. The spatial partitioning data structure can be generated by a k-dimensional tree (KDTree) algorithm.

[0030] In some embodiments, the processor is configured to use computer graphics rendering algorithms, such as ray casting algorithms, to render simulated defect image data.

[0031] Figure 2 A system 200 is shown for generating simulated defect images of component 140 of object 110 for an automated disassembly process. System 200 may include an image capture device 202 and may be combined with means 100 to process the images captured by the image capture device 202. System 200 can be adapted for the automated disassembly of motor 204 using a robotic solution.

[0032] Image capture device 202 may include a 3D scanner and an RGB image capture device (e.g., a camera). A 3D scanner with a dual-mode scanner device includes a 3D scanning device for point cloud data acquisition and an RGB camera for capturing 2D images. In the context of automated disassembly of motor 204, components to be disassembled from motor 204 may include one or more bolts 206.

[0033] In some embodiments, a 3D scanner may be mounted on a robotic arm 208. The robotic arm 208 may be configured to move with at least one rotational motion and at least one translational displacement. One or more platforms (not shown) may be made available for the object 110 to be placed thereon so that a 3D scan can be obtained.

[0034] In some embodiments, the 3D scanner may be a structured light 3D scanner.

[0035] In some embodiments, the robotic arm 208 may be a robotic arm with six degrees of freedom. In some embodiments, the robotic arm may be a six-axis articulated robotic arm and may include one or more controllers for communicating with the processor 150. It will be understood that the robotic arm may have higher degrees of freedom, i.e., more than six degrees of freedom.

[0036] In some embodiments, the robotic arm 208 may be a collaborative robot (cobot), designed for direct human-robot interaction in shared spaces or areas. Such a collaborative robot may be particularly well-suited for situations where the human user / controller and the robot are in close proximity. The collaborative robot may be constructed of lightweight materials, have rounded edges, and may include speed and force limits to enhance safety during operation.

[0037] In some embodiments, the robotic arm may include sensors (e.g., laser displacement sensors) and software to enhance safe behavior during use. For example, robotic arm 208 may be configured to stop operating when it detects an obstacle in its path of motion (e.g., another robot, a human user). In some embodiments, the robotic arm may be equipped with additional components such as force sensors, torque sensors, tool changers, custom mounting frames for mounting 3D scanners, and other robotic actions such as unscrewing components / sensors, diagnosing components, etc.

[0038] In some embodiments, a robotic arm 208 may be provided for disassembling the motor assembly, including but not limited to removing coil windings, differential gears, and loosening external automotive bolts. It can be appreciated that the simplicity and homogeneity of the internal components of the motor may indicate a simpler way to remove the internal components compared to an external housing that is frequently exposed to unknown environmental conditions.

[0039] Figure 2The system 200 depicted can be configured to receive 3D image data 102 of a motor 204 and coordinate information 104 associated with one or more bolts 206. Based on the image data of object 110, a set of component data associated with one or more components 140 can be derived or generated, and relatively realistic (i.e., synthetic) degradation of components, such as bolts, can be overlaid on component 140. In some embodiments, one or more 2D images of object 110 can be captured under different lighting conditions, and components can be manually labeled or annotated for subsequent training of component localization algorithms and / or feature extraction algorithms.

[0040] like Figure 2 The description includes: an edge computing device 150A, which includes a robot control module configured to send control signals to control a robotic arm; and a component detection module configured to use a component localization algorithm to determine the bounding box of each of one or more bolts on the motor, and the set of 3D coordinates of each of the one or more bolts.

[0041] System 200 further includes an edge computing server 150B, which includes an AI-based detection module. The device 100 can be used to periodically or on-demand train the edge server 150B prior to deployment on the edge computer 150A. It will be understood that system 200 can integrate the device 100 into the context of an actual disassembly system solution, where the generated simulated defect images 160 can be used to periodically or on-demand train a vision-based bolt detection AI algorithm deployed on the edge server 150B prior to deployment on the edge computer 150A. In some embodiments, the AI-based module can be trained to perform bounding box annotation using predictions from a pre-trained AI model as annotations. These annotations can be pre-reviewed and corrected by humans and used to train the AI ​​model. In some embodiments, annotation can be performed automatically, i.e., directly using the model's predictions as annotations without review or correction. In some embodiments, crowdsourcing can be used, i.e., distributing data annotation tasks to a large group of people through an online platform.

[0042] Figure 3 A data processing flowchart 300 for an automated synthetic defect generation solution is depicted in stages. Each stage can be in a corresponding subsection. For ease of understanding, object 110 is an electric motor, and the component 140 described for object 110 may include one or more automotive bolts, which can be common in modern vehicles.

[0043] In Phase 1, 3D image data 102 is obtained in the form of 3D Computer-Aided Design (CAD) data 302. Regarding the 3D coordinates 304 of the component, and considering the characteristic complexity of the automotive bolt relative to the motor in 3D space, the center coordinates (i.e., x, y, and z coordinates) of the component may be required. 3D coordinates can be obtained from known sources, such as a list of 3D CAD model components / technical specifications / operation manuals for motors. In some embodiments, a 3D CAD model of the object (e.g., the motor) can be loaded into compatible CAD software. Within the CAD software, a selection tool can be used to precisely locate the automotive bolt on the motor, and the component identifier can be displayed and obtained. For example, the selection tool can precisely locate the bolt with m5bolt_id_X, where the X suffix indicates the bolt's ID. Modern CAD tools allow users to filter out unrelated 3D components based on specified prefixes and / or suffixes. Typical automotive bolts often have cylindrical or hexagonal surfaces. Once the bolt is selected, the corresponding 3D coordinates can be obtained from the properties or information panel within the CAD software. Assuming the top of the bolt has a circular shape, its 3D representation mathematically conforms to the equation of a cylinder. , where r is the radius. The z-axis precisely positions the bolt relative to the defined origin of the model. In modern CAD software, these Cartesian coordinates can be easily determined using measurement or evaluation tools. The aforementioned methodology can be applied to other components (e.g., other bolts) to obtain their coordinates. It is worth noting that some CAD software offers automated listing or export functions, further streamlining the process and saving time.

[0044] Subsequently, the 3D CAD data 302 and the 3D coordinates 304 of component 140 can be output to stage 2 for further processing.

[0045] Phase 2 involves performing pairwise matching of 3D CAD data 302 with bolt coordinate information 304. Due to the complex bolt geometry in 3D space, the component localization algorithm 306 can be configured to perform fine-grained localization for either single or multiple bolts. Additional image fine-tuning / capture can be implemented to capture the relevant 3D bolt surfaces and contours before mapping to a separate identifier for each component, such as a unique ID associated with each bolt.

[0046] Phase 3 may include three modules: an automated feature extraction module 308, a bounding box annotation module 310, and a 3D defect rendering parameter module 312. Based on the output of Phase 2 and the angle of the image capture device 202, the received component information, such as the Cartesian coordinates of the components (e.g., the Cartesian coordinates of the bolts), or Figure 9The target geometry data can be used as input to an automated feature extraction module 308, which automatically filters out unnecessary 3D surfaces, leaving only the bolts. Note that although the image capture device 202 operates in 3D spatial coordinates, the output can be a 2D image—which can be analogous to how a camera works in reality.

[0047] The bounding box annotation module 310 can be configured to receive prior information about the components 140, such as bolt dimensions. The bounding box annotation module 310 may include mathematical operators applied to the 3D coordinates 304, such as the minimum-maximum mathematical operator and simple trigonometry, to automatically obtain the bounding box coordinates of each component 140.

[0048] The 3D defect rendering parameter module 312 can be configured to render corresponding defect types and intensities on the bolt surface. The defect type and intensity can be based on a first image parameter and a second image parameter. Examples of rendered defects are rust, scratches, and color. In some embodiments, at least one of the first and second parameters can be normalized to a range of 0 to 1.

[0049] In stage 4, the output from modules 308, 310, and 312 can be input into the 3D rendering engine 314 and the UI defect control 316. The 3D rendering engine 314 can be configured to overlay the visual effects received from stage 3 onto one or more components 140. To check the realism of the rendered defects, the user can activate the UI defect control 316 using a key combination such as "Ctrl + Shift + F". In some embodiments, the key combination is user-configurable. The defect type and intensity can be changed by the user, with updated changes reflected in real time. Similarly, the UI defect control can be used to customize defects on individual bolts.

[0050] In stage 5, the user may need to determine the number of paired images and annotations to be generated as input to the generator application. For example, a predetermined number of samples, such as 500 or 5000 image samples with different environmental parameters (such as rust, scratches, and lighting intensity), can be generated and pre-stored in a database. Afterward, the paired data is output to a folder where the user can retrieve the images and import them as a training dataset for the desired object detection model.

[0051] In some embodiments, additional parameters may be used for minor adjustments or fine-tuning. Such additional parameters include, but are not limited to, camera angle and external light source illumination angle to simulate actually changing lighting conditions, which may affect the accuracy of any object detection model.

[0052] The output from stage 5 degenerated includes output set 318, which includes 2D simulated defect images 160 of bolts 206 and bounding box annotations around each bolt 206.

[0053] In the various embodiments described, the component localization algorithm may include a spatial partitioning data structure configured to receive the set of 3D coordinates of component 140 as input and output the set of component data. For example... Figure 4 As illustrated in [2][3], the k-tree (KDTree) algorithm [1] can be used to generate spatial partitioning data structures. In some embodiments, the k-tree (KDTree) algorithm can locate components and use real-world textures to graphically render specific defects. In particular, the use of the ray casting algorithm [4] can be found in Blender [5], a 3D computer graphics software used to create visual effects, animated films, and video games.

[0054] It should be understood that data labeling / annotation of target objects can be automated with relatively high accuracy, while improving human productivity for highly repetitive tasks of data annotation.

[0055] In some embodiments, the KDTree algorithm can be configured to efficiently represent data structures of nodes in a multidimensional space. It is widely used in computer graphics, machine learning, and computational geometry for tasks such as nearest neighbor search and range search. A kd-tree represents points as nodes in a binary tree.

[0056] A tree can be constructed recursively by partitioning elements along different dimensions of space. At each level, the partitioning dimensions change, resulting in a balanced tree; see [link to documentation]. Figure 4 Illustration.

[0057] To construct a tree, you can select a set of points and choose / select a dimension to partition them. For example... Figure 4 As shown, elements can be divided into two subsets based on the median along this dimension. This procedure can be repeated recursively until the termination criterion is met for each subset, and facilitates efficient search point queries when the tree structure is completed [3]. Then, nearest neighbor search can be used to traverse the tree structure by comparing points along the partitioning dimension and selectively examining relevant subsets to limit the search space. The nearest points found are frequently maintained and updated whenever a closer point is identified.

[0058] In some embodiments, range searches can be performed by selectively examining subsets of trees that intersect with a specified range, thereby efficiently identifying points within that range. In this case, it can be inferred that no two points share the same 3D coordinates (i.e., x, y, z). Therefore, the kd-tree algorithm can be considered an efficient method for organizing points in multidimensional space, facilitating fast and accurate queries. In particular, the recursive nature and partitioning strategy of KDTrees make them suitable for a wide variety of research fields, thereby enhancing performance and facilitating advanced data analysis.

[0059] In the context of motor disassembly, the KDTree method can be applied to each bolt of interest using its central 3D coordinates, and this KDTree method is applied to visually observable adjacent surfaces (or relevant vertices). The mapping of the KDTree output to each bolt is obtained.

[0060] In some embodiments, defect rendering algorithms may include Blender software. Blender software offers a wide variety of tools and techniques for creating defects in rendered images, such as texture mapping, procedural shaders, or post-processing effects. Defect rendering in Blender is the intentional addition of imperfections to create more vivid and lifelike virtual environments. By simulating defects or degradation, such as scratches, dust, and other natural artifacts, users can add depth and realism to their digital creations.

[0061] In some embodiments, tools such as texture mapping and procedural shaders can enhance the appearance of objects and materials. Figure 5 The figure shows rust renderings of different intensities on all surfaces of the automotive bolts, all of which can be simulated using the Perlini noise function [5]. Based on the output of KDTree, bolt defects of different intensities can now be uniformly applied to either one or more bolts.

[0062] In addition, users can select to randomly apply defects, such as scratches, to a number of bolts at once. Figure 6 As shown.

[0063] In some embodiments, the UI defect control 316 can be implemented as a slider control for dynamic visualization of wear and damage effects, such as... Figure 7 As shown.

[0064] Figure 7 The proposed slider-based user interface (UI) shown in the figure can provide users with an intuitive and seamless experience, regardless of their technical expertise. The UI can present data in a relatively easy-to-understand way, enabling the rapid simulation of complex real-world conditions and the extraction of desired defect parameter ranges.

[0065] In some embodiments, users can simply input the required defect parameters, individual parameter ranges (if applicable), and number of images into the UI defect control 316 to generate 2D image results and streamline the entire process.

[0066] In some embodiments, a high-level programming language such as Python can be used to program the UI defect controls.

[0067] In some embodiments, the bounding box annotation module 310 may be configured to annotate components, such as partially occluded components, based on a ray casting algorithm [4]. A ray casting algorithm is a tool used for rendering 3D images before utilizing the computing power of a graphics processing unit for ray tracing immersion. References Figure 8 For each pixel of the rendered image 802, a ray can be projected onto an object, and the intersection point 804 of the ray with each object's face (i.e., surface) is calculated. Given the camera's intrinsic parameters, the direction of the ray projected through the pixel (x, y) in the rendered image can be calculated, and the ray's direction vector can be mathematically represented by the following equation (1): Where i, j, It is a vector of camera intrinsic parameters corresponding to focal length, aperture, field of view, and resolution, and light... The origin is the camera position. Therefore, light rays can be represented by parametric equations. , where d is the direction vector of the ray in equation (1), and It is camera location 806, see [link / reference] Figure 8 Each facet of the object that intersects with the light rays projected by camera 808 can be represented by an implicit equation. Where n is the 3D normal vector of the object's face, P is the 3D vector coordinate of a point on the face, and D is the distance from the face to the origin. Solving the system of equations allows us to find the intersection point and verify whether the intersection point lies within the face, and whether t is within [t_min, t_max]. It can be understood that the 3D mesh can be viewed as a combination of faces (planes), and therefore pixels belonging to the object can be assigned based on the nearest intersection point.

[0068] like Figure 9 As shown, ray projection 902 can provide or output a segmentation map 904 of the rendered image. By using the geometric data (e.g., 3D coordinates) of the target of interest 906 provided from the scanning step, then parameters such as... can be extracted for each target object. Figure 9The shown image contains the mask information for a cube or cuboid. The segmentation mask 908 of the cube or cuboid contains pixel coordinates and can be represented as two one-dimensional (1D) arrays, i.e. and Combining Figure 4 As can be seen, using the KDTree algorithm, the minimum and maximum values ​​of each 1D array can be extracted to infer the bounding box coordinates of the object. In other words, the bounding box coordinates can be derived from the segmentation mask.

[0069] Although ray casting algorithms are simple, they are computationally expensive, and their complexity increases with image resolution and the number of primitive surfaces in object space. For example, for an image resolution of 1000 pixels by 1000 pixels, a CAD model has at least one million surfaces and one million rays (i.e., Rays can be projected. Blender software promotes efficiency by introducing accelerated structures, such as Bounded Volume Hierarchy (BVH)[6]. These structures can be constructed using the 3D geometry of objects visible in the scene. When projecting rays, this structure enables the removal of unwanted surfaces and saves computational costs.

[0070] According to another aspect and reference Figure 10 A computer-aided method 600 is provided for generating simulated defect images of components 140 of object 110 for an automated disassembly process. The method 600 includes the following steps: obtaining 602 three-dimensional (3D) image data 102 of object 110 and a set of 3D coordinates of components 140 of object 110; determining 604 a set of component data based on the 3D image data 102 and the set of 3D coordinates of component 140 using a component localization algorithm; filtering 606 non-component image data from the 3D image data 102 based on the set of component data as input to a feature extraction algorithm to obtain filtered 3D image data; rendering 608 simulated defect image data on the filtered 3D image data; and generating 610 a simulated defect image 160 based on the rendered simulated defect image data. In some embodiments, the simulated defect image 160 includes a 2D image and annotation data. In some embodiments, the method 600 may be implemented as a computer program mounted on a processor 150, the computer program including instructions for performing the method 600. The filtered 3D image data may be extracted component image data.

[0071] References [1] Bentley, JL, 1975. Multidimensional binary search trees for association search. ACM Communications, 18(9), pp. 509-517.

[0072] [2] Anzola, J., Pascual, J., Tarazona, G. and Gonzalez Crespo, R., 2018. Clustered WSN routing protocol based on kd-tree algorithm. Sensors, 18(9), p. 2899.

[0073] [3] Kakde, HM, 2005. Range search using kd-trees. Florida State University.

[0074] [4] Scott D Roth, Ray casting for modeling solids, Computer Graphics and Image Processing, Vol. 18, No. 2, 1982, pp. 109-144, ISSN 0146-664X, https: / / doi.org / 10.1016 / 0146-664X(82)90169-1.

[0075] [5] Community, BO, 2018. Blender - a 3D modeling and rendering software package, Stichting Blender Foundation, Amsterdam. Available at: https: / / docs.blender.org / manual / en / 3.6 / render / freestyle / view_layer / line_style / modifiers / geometry / perlin_noise_2d.html.

[0076] [6] Gunther, J., Popov, S., Seidel, HP and Slusallek, P., September 2007. Real-time ray tracing on GPUs based on BVH package traversal. IEEE Interactive Ray Tracing Workshop 2007 (pp. 113-118). IEEE.

Claims

1. An apparatus (100) for generating simulated defect image data (160) of a component (140) of an object (110) for an automated disassembly process, the apparatus (100) comprising a processor (150) configured to: Obtain three-dimensional (3D) image data (102) of the object (110) and a set of 3D coordinates (104) of the components (140) of the object (110). Using a component localization algorithm, a set of component data is determined based on the 3D image data (102) and the set of 3D coordinates (104) of the component (140); Based on this set of component data, a feature extraction algorithm is used to filter non-component image data from the 3D image data (102) to obtain filtered 3D image data; and Simulated defect image data is rendered on the filtered 3D image data to generate simulated defect image data (160).

2. The apparatus (100) according to claim 1, wherein, The simulated defect image (160) includes two-dimensional (2D) image data and annotation data associated with the component (140).

3. The apparatus (100) according to claim 1 or 2, wherein, The device includes a graphical user interface (GUI) (170) configured to display a set of parameters (172, 174) associated with the simulated defect image data for user input.

4. The apparatus (100) according to any one of the preceding claims, wherein, The component data is obtained by comparing the set of 3D coordinates (104) of the component (140) with data from a trained component database.

5. The apparatus (100) according to any one of the preceding claims, wherein, The component localization algorithm includes a spatial partitioning data structure configured to receive the set of 3D coordinates of the component (140) as input and output the set of component data.

6. The apparatus (100) according to claim 5, wherein, The k-dimensional tree (KDTree) algorithm is used to generate the spatial partitioning data structure.

7. The apparatus (100) according to any one of the preceding claims, wherein, The processor is configured to use computer graphics rendering algorithms to render the simulated defect image data.

8. The apparatus (100) according to claim 7, wherein, The computer graphics rendering algorithm is a ray casting algorithm.

9. An automated dismantling system (200), the system (200) comprising the apparatus (100) according to any one of the preceding claims, wherein, The simulated defect image data (160) output from the device (100) is configured as training input to train a vision-based component detection AI algorithm.

10. The system (200) of claim 9, further comprising a 3D scanner and an RGB image capture device (202), the 3D scanner and the RGB image capture device (202) being configured to obtain three-dimensional (3D) image data of the object (110) and a set of 3D coordinates (104) of components of the object (110), wherein, The object (110) is an electric motor (204), and the component (140) includes one or more bolts (206) of the electric motor (204), wherein the system (200) includes a robotic arm (208) operable to remove the one or more bolts (206), and wherein the 3D scanner and the RGB image capture device (202) are mounted on the robotic arm (208).

11. The system (200) according to claim 10, wherein, The system (200) further includes an edge computing server (150B), which includes an AI-based detection module.

12. The system (200) according to claim 10 or 11, wherein, The robotic arm (208) is arranged to communicate with an edge computing device (150A), the edge computing device (150A) including... A robot control module configured to send control signals to control the robotic arm; as well as A component detection module is configured to use the component localization algorithm to determine the bounding box around each of one or more bolts on the motor, and the set of 3D coordinates of each of the one or more bolts.

13. A computer-aided method (600) for generating simulated defect image data of components (140) of an object (110) in an automated disassembly process, the method (600) comprising the following steps: Obtain (602) three-dimensional (3D) image data (202) of the object (110) and a set of 3D coordinates of the components (140) of the object (110); Using a component localization algorithm, a set of component data (604) is determined based on the set of 3D coordinates of the 3D image data (202) and the component (140); Based on the set of component data as input to the feature extraction algorithm, non-component image data is filtered (606) from the 3D image data (202) to obtain filtered 3D image data; Render (608) the simulated defect image data on the filtered 3D image data; as well as A simulated defect image (160) is generated (610) based on rendered simulated defect image data, wherein the simulated defect image (160) includes one or more component annotation data.

14. The computer-aided method according to claim 13, wherein, The simulated defect image includes two-dimensional (2D) image data and annotation data that are automatically associated with the corresponding component (140).

15. A computer program comprising instructions for performing the computer-aided method according to claim 13 or 14.

Images