System and method for estimating a health state value of a product for a disassembly process

Abstract

The present invention relates to a system (100) for estimating the health status value (130) of a product (110) for a disassembly process, the system (100) including a processor (150) configured to: acquire image data (120) of the product (110); identify product data (121, 122) of the product (110) based on the image data (120) using a first machine learning algorithm; and identify components (140) of the product (110) based on product type using a second machine learning algorithm, and output component identifiers (140) of the components (140). 23) and three-dimensional location (124); assign material type (125A, 125B) to component (140); use a third machine learning algorithm to identify defects on component (140) based on material type (125A, 125B), component identifier (123) and three-dimensional location (124); classify the identified defects based on defect parameters (128) of the identified defects to obtain classified defects (129), and estimate the health status value (130) of component (140) based on the classified defects (129) on component (140).

Description

Technical Field

[0001] This disclosure relates to systems and methods for estimating the health status values ​​of products during disassembly processes. Background Technology

[0002] Material recycling / reuse is an important consideration in the higher-level goals of the circular economy. By dismantling old products that have reached the end of their service life, usable components can be extracted from these old products (which may include one or more components). Depending on the quality, the dismantled components can either be reused or recycled.

[0003] The process of disassembling a specific product has its own costs, which can be in terms of the energy, time, and resources spent on the disassembly process. On the other hand, the value that can be extracted from a product typically depends on the quality of the components and their material type. For a given material type, its potential value for reuse / recycling depends on the quality of said material, which in turn is affected by the level of degradation of (multiple) components.

[0004] Currently, there is a lack of methods or systems for determining product value during disassembly. Furthermore, there is a need to optimize the cost-return relationship in product disassembly. Summary of the Invention

[0005] This disclosure is conceived as a technical solution for providing an estimate of the health status of a product (and one or more components thereof). The health status of the product can be represented as a value, with lower values ​​associated with a lower health status and higher values ​​associated with a higher health status. In some embodiments, the health status value can form part of a decision regarding whether the product should be disassembled or otherwise disposed of. This decision can, in turn, be based on an optimization objective function that maximizes the potential value that can be extracted from the product when it is disassembled.

[0006] In some embodiments, this disclosure can form part of a disassembly sequence planning (DSP) for disassembling a product comprising an assembly of one or more components, wherein the health status of the individual components in the assembly (which may be associated with defect status) is determined using a machine learning model and taken into account in the DSP. One or more defects on the components can be detected, and the detected defects can be classified with respect to their intensity / severity level.

[0007] According to one aspect of this disclosure, a system as claimed in claim 1 is provided. According to another aspect of this disclosure, claim 8 defines a computer-aided method according to this disclosure. Claim 10 defines a computer program comprising instructions for performing the computer-aided method.

[0008] The dependent claims define several examples relating to the system and the method, respectively. Attached Figure Description

[0009] This disclosure will be better understood with reference to the detailed description when considering non-limiting examples and the accompanying drawings, wherein: - Figure 1A This is a schematic diagram of an embodiment of a system for estimating the health status values ​​of a product during the disassembly process; - Figure 1B This is a schematic diagram illustrating the data flow between various modules of a system according to some embodiments; - Figure 2A The image shows a product shaped like a washing machine with broken glass and damaged plastic components. - Figure 2B The illustration shows dirt and rust on a product shaped like an electric drill. - Figure 3 Examples of bolt assemblies for products illustrated in Figures A, 3B, and 3C, along with estimates of defect severity and health status; and - Figure 4 This is a flowchart of a method for estimating the health status of a product during the disassembly process. Detailed Implementation

[0010] The following detailed description refers to the accompanying drawings, which illustrate specific details and embodiments in which the present disclosure may be practiced. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the present disclosure. Other embodiments may be utilized, and structural and logical changes may be made, without departing from the scope of the present disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.

[0011] The embodiments described in the context of one of the systems or methods are similarly effective for other systems or methods.

[0012] Features described in the context of one embodiment can be applied accordingly to the same or similar features in other embodiments. Features described in the context of one embodiment can be applied accordingly to other embodiments, even if not explicitly described in those other embodiments. Furthermore, additions and / or combinations and / or substitutions described for a feature in the context of one embodiment can be applied accordingly to the same or similar features in other embodiments.

[0013] In the context of various embodiments, the articles “a,” “an,” and “the” used with respect to a feature or element include references to one or more of the features or elements.

[0014] As used in this article, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0015] As used herein, the term "data" can be understood to include information in any suitable analog or digital form, such as information provided as a file, part of a file, a collection of files, a signal or stream, a part of a signal or stream, a collection of signals or streams, etc. However, the term data is not limited to the examples above, but can take various forms and represent any information as understood in the art.

[0016] As used herein, the term "module" refers to or forms part of or includes the following: application-specific integrated circuits (ASICs); electronic circuits; combinational logic circuits; field-programmable gate arrays (FPGAs); processors (shared, dedicated, or grouped) that execute code; other suitable hardware components that provide the described functionality; or combinations of some or all of the foregoing, such as in a system-on-a-chip. The term "module" may include memory (shared, dedicated, or grouped) that stores code executed by a processor.

[0017] As used in this article, the terms “first,” “second,” “third,” “fourth,” etc., are used for clarity purposes and do not imply order or priority.

[0018] As used herein, the term "disassembly process" refers to the process of separating a product into its components. In some embodiments, a disassembly process can be used to separate one or more components (e.g., bolts or other fasteners) from a product (e.g., an electric motor, a washing machine) and / or a partial assembly via a non-destructive or semi-destructive process, possibly targeting connectors / fasteners. Such disassembly processes can be applied to a variety of industries, particularly but not limited to waste treatment facilities, reverse engineering processes, etc. Disassembly processes can broadly include guided disassembly processes using augmented reality, virtual reality, and / or mixed reality; automated disassembly (i.e., fully automated).

[0019] As used herein, the term "processor" refers to a circuit, including analog, digital, or mixed-signal circuits, or components thereof. According to alternative embodiments, any other kind of implementation of the various functions described in more detail below may also be understood as a "circuit." A digital circuit can be understood as any kind of logical implementation entity, which may be a dedicated circuit or a processor or firmware that executes software stored in memory.

[0020] As used herein, the term “acquisition” refers to a processor that actively acquires input or passively receives input from a user interface and / or one or more sensors. The term “acquisition” can also refer to a processor that receives or acquires input from a communication interface, such as a user interface. Processors or dosing modules may also receive or acquire input via memory, registers, and / or analog-to-digital ports.

[0021] As used herein, the term "product" includes a complex body comprising at least one component or one or more components. Non-limiting examples of products include electric motors (components including one or more bolts), washing machines, electric drills, etc.

[0022] Figure 1A and Figure 1B An embodiment of the present disclosure is shown, illustrating the setup of a system 100 for estimating the health status value of product 110 during a disassembly process.

[0023] System 100 may include processor 150, configured to acquire image data 120 of product 110; identify the product type of product 110 based on the image data 120 using a first machine learning algorithm; identify one or more components of product 110 based on the product type using a second machine learning algorithm, and output a component identifier and three-dimensional position for each of the one or more components; assign a material type to each of the one or more components; identify defects on each of the one or more components based on the material type, component identifier, and three-dimensional position using a third machine learning algorithm; classify the identified defects based on defect parameters of the identified defects; and estimate a health status value of one or more components 140 based on the defect parameters for each of the one or more components. In some embodiments, the defect parameters may be the severity and / or intensity of the defect. For example, for components containing iron, the defect parameters may be a percentage of rust, such as 15% rust, 45% rust, or 80% rust.

[0024] System 100 can be part of an optimized system for an automated dismantling system / plant of a Material Recycling Facility (MRF). The optimized system can be implemented upstream of the dismantling system to determine whether product 110 should be dismantled or shredded / discarded.

[0025] Referring again to Figure 1, processor 150 may include a product identification module 152, a component positioning module 154, a component material information module 156, a defect identification module 158, and a defect classification module 160. System 100 may include a disassembly decision module 170, configured to determine whether product 110 should be disassembled or scrapped based on output from defect classification module 160. Defect identification module 158 and defect classification module 160 may be part of a health status estimator module.

[0026] System 100 may include an image capture device 180 for capturing image data 120 of product 110. In some embodiments, the image capture device 180 may include an RGB image capture device (e.g., a camera). The RGB image capture device may be configured to capture two-dimensional (2D) images in various formats, such as JPEG, bitmap, etc. In some embodiments, the image capture device 180 may include a 3D scanning device for point cloud data collection. In some embodiments, the image data acquired by the 3D scanning device may be converted into computer-aided design (CAD) files.

[0027] Product identification module 152 can be configured to acquire image data 120 of product 110 (e.g., image data of an electric motor) as input. A first machine learning algorithm is configured to estimate product data / information, such as product type 121 and its specific model 122, as output data. An example of the first machine learning algorithm can be a deep neural network (DNN), such as a convolutional neural network. In other words, product identification module 152 can be considered a product classifier and identifies the exact model / type of product 110. It can be understood that estimating product information (such as product type) may be important because the disassembly sequence and specific disassembly instructions between products may differ. In some embodiments, the first machine learning algorithm is a first deep neural network (DNN). Depending on the type of product to be disassembled, the first DNN may be trained on a combination of publicly available data, such as from a publicly available image database, such as the ImageNet database [1], and product-specific datasets.

[0028] The component positioning module 154 can be configured to receive product type 121 and / or specific product model data 122 as input in order to identify and locate one or more components of product 110. This can be based on the disassembly diagram sequence.

[0029] In some embodiments, the identification and localization of one or more components of product 110 may include at least one of the following methods: (a) synchronizing image data 120, such as 3D CAD data acquired from a 3D scanner, with a 3D scanned model; and (b) training a second machine learning algorithm, which may be a second DNN, on all components specific to product 110. The expected output data of component localization module 154 may be a component name (e.g., a bolt) 123 and its 3D location or position 124 relative to product 110 (e.g., represented in 3D coordinates), for example, in a coordinate system assigned to or attached to product 110.

[0030] The component material information module 156 can be configured to acquire material-specific data 125A of the product 110 and its components. It will be understood that the health status parameters can vary in nature for each component of the product 110. For example, a particular component (e.g., the door of a side-load washer or dryer) may be made of glass or plastic, which may be more prone to breakage / scratching than rusting. Similarly, components made of metal, such as iron (e.g., bolts, screws), may be more prone to rusting, deformation, etc., but less prone to complete breakage. In some embodiments, the information 125A relating to the material or substance of each component of the product 110 can be used for subsequent health status value determination. In some embodiments, the information 125A relating to the material or substance of each component can be acquired either by (a) using electromagnetic (EM) spectroscopy and computer vision-based methods or (b) by using 3D model data, such as CAD data, and bill of materials analysis. Such information relating to the material or substance of each component can be stored in one or more databases 190. Material-specific data 125A can be processed by component material information module 156 to obtain relevant data entries 125B suitable for input into defect identification module 158.

[0031] The defect identification module 158 and the defect classification module 160 can form part of the health status estimator module. It can be understood that the health status of a given product / component can be roughly estimated based on two types of defects: functional defects and visual defects.

[0032] Functional defects can involve whether a given component of product 110 is in a working state. Functional defects may not be easily identifiable by the naked eye.

[0033] Visual defects can involve the physical characteristics of 110 components of a product. Non-limiting examples include those visible to the naked eye, such as shape, size, scratches, etc.

[0034] In some embodiments, the defect identification module 158 is configured to focus on visual defects to determine a health status value, which is then used for health status estimation. The defect identification module 158 may be configured to use a third machine learning algorithm, such as a third DNN, to detect and identify material-specific defects. In this regard, the defect identification module 158 may be configured to receive output data 123, 125 from the component location module 154 and processed material-specific data 125B output from the component material information module 156. Based on the component name (e.g., bolt), 3D location or position, and component material information, one or more visual defects on each of one or more components can be identified.

[0035] In some embodiments, the defect identification module 158 may be trained to identify common defects (multiple) for each material type as well as product / component-specific defects. For example, while plastic components of two different types of products may share some common defects, they may also exhibit different defects depending on how and where the products are used.

[0036] Figure 2A and Figure 2B Two examples of potential defects in some commonly used products 110 are shown. Figure 2A A product 110 in the form of a washing machine 110A is shown. The washing machine 110A may include one or more defects, such as broken glass 202A or broken plastic components 202B. Figure 2B Product 110 is shown in the form of electric drill 110B. Electric drill 110B may include one or more defects in the form of dirt and / or rust 204 on (multiple) surfaces or various components of electric drill 110B.

[0037] It can be understood that the third machine learning algorithm can be trained to identify various defects 202A, 202B, and 204. The output data of the defect identification module 158 may include image data of the identified component 126 (e.g., bolt) with defect type 127 (rust).

[0038] Image data of the identified component 126, including defect type data 127, can be used to derive defect parameters 128. Defect parameters 128 can indicate the relative intensity or severity level of the defect. A product / component with several high-severity defects can be considered to have a low health status value, and vice versa.

[0039] The image data of the identified component 126 and the defect type data 127 can be used as input data to the defect classification module 160. The defect classification module 160 can then generate classified defects 129. The classified defects 129 can be used to derive health status values ​​130.

[0040] Figure 3 A to Figure 3 C shows three examples of components in the form of bolt 302 that belong to different health status categories depending on the strength or severity of defects (such as corrosion). Figure 3 A shows bolt 302A with a small amount of rust, which will be classified as corresponding to a relatively high health condition value. Figure 3 B shows bolt 302B with moderate rust, which will be classified as corresponding to a moderate health condition value. Figure 3 C illustrates a severely corroded bolt 302C, which will be classified as corresponding to a low health status value. In each of the above examples, computer vision techniques can be used to classify the level of defect severity. In some embodiments, a fourth machine learning algorithm, such as a fourth DNN, can be trained as an image classifier that categorizes a given image of a defective component into various severity categories. Depending on the health status category value, different dismantling decisions can be made. For example, a component with the lowest health status value may be scrapped directly rather than further dismantled. On the other hand, a component with the highest health status value should be completely dismantled to extract the maximum value from that component.

[0041] In some embodiments, the health status value can be standardized in the range of 0 to 1, where 0 corresponds to the lowest health status value and 1 corresponds to the highest health status value.

[0042] In some embodiments, product 110 includes multiple components, and each component is considered to have a different health status value, and can obtain overall statistical measures such as average health status value, median health status value, and mode health status value.

[0043] The disassembly decision module 170 can be configured to receive product information, such as information obtained from a bill of materials, and health status values ​​(or statistical measures of multiple health status values), as input. The disassembly decision module 170 can then be configured to perform an assessment of the material value of the components, and can then calculate the components of the partial assembly. The cost of disassembling product 110 can then be calculated. Optimization of this decision can include one or more parameters, such as the benefits from materials, which may in turn be affected by the health status of the components(s). In other words, the worse the health status of the components(s) of product 110, the more likely product 110 will not be disassembled and can instead be scrapped or otherwise disposed of.

[0044] In some embodiments, the optimization process can be modeled as a mixed-integer linear programming problem with an objective function that maximizes the profit from disassembly. Alternatively, the objective function can be modeled as minimizing the cost from disassembly.

[0045] In some embodiments, the health status estimator can be configured to provide material information about the component. This material information is then used to determine the path to be taken for disassembly. This process can then be repeated until the product is disassembled to a profitable extent.

[0046] In various embodiments, supervised learning, unsupervised learning, or hybrid learning models can be used to train at least one of the first, second, third, and fourth machine learning algorithms. It will be appreciated that one or more image databases, including labeled 2D or 3D images of different products 110, components, and defect severity / intensity, can be used to facilitate the training process.

[0047] In various embodiments, a first machine learning algorithm, a second machine learning algorithm, a third machine learning algorithm, and / or a fourth machine learning algorithm can be combined. Such combinations may include, for example, a combination of the first and second machine learning algorithms, and / or a combination of the third and fourth machine learning algorithms. Other permutations and combinations of various machine learning algorithms are conceivable.

[0048] In some embodiments where product 110 undergoes a disassembly phase, i.e., a phased disassembly of components, the proposed system 100 can be configured to recursively use the output data of the defect identification module 158 and the defect classification module 160 as input data to the disassembly decision module 170 to make an optimal disassembly decision for each component. In other words, the health status estimation of the components and their corresponding optimal disassembly actions occur recursively until the product is completely disassembled or scrapped. Figure 1A As shown, after the disassembly decision module 170 has decided to disassemble component 140, the disassembly decision module can send a signal to the component positioning module 154 to identify the next component 140 to be disassembled.

[0049] In some embodiments, the automated disassembly process can be used for disassembling the electric motor and may include removing one or more components 140, such as one or more bolts, from the electric motor. In some embodiments, the automated disassembly process may be part of a waste or material recycling facility, such as an electronic waste recycling facility.

[0050] In some embodiments, the component localization algorithm includes a spatial partitioning data structure configured to receive a set of 3D coordinates of component 140 as input and output a set of component data. The spatial partitioning data structure can be generated using a k-dimensional tree (KDTree) algorithm.

[0051] According to another aspect and reference Figure 4A computer-aided method 400 is provided for estimating the health status value of product 110 during disassembly, the method 400 comprising the following steps: Step 402: Acquire image data 120 of product 110; Step 404: Using the first machine learning algorithm, identify the product type of product 110 based on image data 120; Step 406: Using a second machine learning algorithm, identify one or more components 140 of product 110 based on product type, and output the component identifier and three-dimensional position of each of the one or more components; Step 408: Assign a material type to each of the one or more components 140; Step 410: Based on material type, component identifier, and three-dimensional location, use a third machine learning algorithm to identify defects on each of the one or more components; Step 412: Classify the identified defects based on their defect parameters, and Step 414: Estimate the health status value of the product based on the defect parameters for each of the one or more components.

[0052] In some embodiments, method 400 may be implemented as software executable instructions or a computer program.

[0053] According to another aspect of this disclosure, a method for disassembling a product using a disassembly diagram model is provided. For a given product 110 to be disassembled, a disassembly diagram, including nodes (where each node represents a component 140 to be disassembled) and leaf nodes indicating the sequence of disassembly instructions to be followed, can be based on a process that may be referred to as health-state driven disassembly: a. Take node N from disassembly diagram G. b. Identify and locate the product sub-component represented by N in the actual product. c. Estimate N's health status. d. Estimate the optimal disassembly decision for N and execute that disassembly decision. e. Repeat step ad until the entire product is disassembled.

[0054] According to another aspect of this disclosure, there exists a computer program that includes instructions for performing a computer-aided method. In some embodiments, a non-transitory computer-readable medium configured to store executable software instructions thereon may be included, such that when executed, method 400 is performed.

[0055] refer to [1] Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, SanjeevSatheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, MichaelBernstein, Alexander C. Berg and Li Fei-Fei. (* = equal contribution)ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015.

Claims

1. A system (100) for estimating a health status value (130) of a product (110) during a disassembly process, the system (100) comprising a processor (150) configured to: Acquire image data (120) of product (110); Using a first machine learning algorithm, product data (121, 122) of the product (110) are identified based on the image data (120). Using a second machine learning algorithm, the components (140) of the product (110) are identified based on the product type, and the component identifier (123) and three-dimensional position (124) of the component (140) are output. Assign material types (125A, 125B) to the component (140). Using a third machine learning algorithm, defects on the component (140) are identified based on the material type (125A, 125B), the component identifier (123), and the three-dimensional position (124); The identified defects are classified based on the defect parameters (128) of the identified defects to obtain the classified defects (129), and The health status value (130) of the component (140) is estimated based on the classified defects (129) on the component (140).

2. The system (100) according to claim 1, further comprising a disassembly decision module (170) configured to provide an indication of whether to disassemble the component (140) from the product (110) based on the health status value (130).

3. The system (100) of claim 2, wherein the disassembly decision module (170) is configured to send feedback data to the processor (150) to use the third machine learning algorithm to identify defects on another component of the product (110) after estimating the health status value (130) of the component (140).

4. The system (100) according to any one of the preceding claims, wherein, The processor (150) is configured to classify the identified defects using a fourth machine learning algorithm.

5. The system (100) according to any one of the preceding claims, wherein, At least one of the first machine learning algorithm, the second machine learning algorithm, the third machine learning algorithm and the fourth machine learning algorithm is a deep neural network (DNN).

6. The system (100) according to any one of the preceding claims, wherein, The product (110) includes multiple components (140), and the processor (150) is configured to calculate a statistical measure of the health status value (130) of the multiple components (140).

7. The system (100) according to any one of the preceding claims further includes a product information database (190) arranged to communicate with the processor (150) to assign the material type (125A, 125B) to the component (140).

8. A computer-aided method (400) for estimating the health status value (130) of a product (110) during a disassembly process, the method (400) comprising the steps of: Acquire (402) the image data (120) of the product (110); Using a first machine learning algorithm, product data (121, 122) of the product (110) are identified (404) based on the image data (120). Using a second machine learning algorithm, the components (140) of the product (110) are identified (406) based on the product data (121, 122), and the component identifier (123) and three-dimensional position (124) of each component in the component (140) are output. Assign (408) material types (125A, 125B) to the component (140). Based on the material type (125A, 125B), the component identifier (123), and the three-dimensional position (124), a third machine learning algorithm is used to identify (410) defects on the component (140); The identified defects are classified (410) based on the defect parameters (128) of the identified defects to obtain the classified defects (129), and The health status value (130) of the component (140) is estimated (412) based on the classified defects (129) on the component (140).

9. The method (400) of claim 8, further comprising the step of providing an indication of whether the component (140) should be removed from the product (110) based on the health status value (130).

10. A computer program comprising instructions for performing the computer-aided method according to claim 8 or 9.

Images