System and method for augmenting data to estimate a state of a product

Abstract

Aspects concern a system (100) for augmenting data to estimate a state of a product (110), the system (100) includes a processor (132), the processor (132) configured to: obtain a first set of image data (120) of the product (110), identify, using a product identification module (134), a product data (121, 122) of the product (110) based on the first set of image data (120), retrieve a usage data (123, 124) of the product (110) based on the product data (121, 122), assign a material type (125) to the product (110), and predict, using a generative artificial intelligence (AI) based model (140), a second set of image data (150) of the product (110) based on the material type (125), the usage data (123, 124) and the product data (121, 122). The second set of image data (150) is associated with the first set of image data (120) based on a period of use of the product (110).

Description

SYSTEM AND METHOD FOR AUGMENTING DATA TO ESTIMATE A STATE OF A PRODUCTTECHNICAL FIELD

[0001] This disclosure relates to a system and a method for augmenting data to estimate a state of a product.BACKGROUND

[0002] Prolonged product usage and aging inevitably leads to damages (i.e., wear and tear) to a product. By understanding how the product undergoes wear and tear, useful information can be extracted for industrial applications, such as automatic predictive maintenance, automatic defect identification and spare-part identification, which may lead to reduction in repair costs and unexpected downtime from product failure.

[0003] Conventionally, the industrial applications rely on a deep neural network (DNN) model in combination with industrial internet of things (IIoT). The accuracy of a DNN backed system mainly depends on the specific DNN architecture being used and the quality of dataset for training the DNN model. While there are many publicly available DNNs, such as YoLo, SSD and PointNet, it is important to train these models using a high-quality and large-scale dataset. However, gathering a large amount of high-quality data can be a laborious task, especially for products which do not have any publicly available datasets.

[0004] One of the conventional techniques that effectively increases a dataset size is data augmentation which involves random cropping, orientating and distorting images. However, such techniques do not take into account real-time variations in the condition of a used product due to usage and age. As such, traditional data augmentation techniques would not be sufficient to train DNN models to identify used products.

[0005] Thus, there is at present a lack of method or system for augmenting data to estimate a state of a product.SUMMARY

[0006] This disclosure was conceptualized to provide a technical solution for augmenting data using real-life defects in actual products to estimate a state of a product (and one or more components of the product). Based on an input image of the product at a time T, the product is identified and the relevant material information and usage pattern information are obtainedfrom one or more databases. An estimation on the state of the product may be generated using a generative artificial intelligence (Al) model. The estimation may be presented as a set of image data which includes an output image of the product at a time T+A, where A represents the time for which the product has been used. The output image represents how the product would look like after being used for A amount of time in a given condition, including possible defects on the product due to prolonged usage and aging. In some embodiments, the output image may be incorporated into a dataset for training the generative Al model.

[0007] According to an aspect of the present disclosure, a system as claimed in claim 1 is provided. According to another aspect of the present disclosure, a computer-assisted method according to the disclosure is defined in claim 10. A computer program comprising instructions to execute the computer-assisted method is defined in claim 12.

[0008] The dependent claims 2 to 9 define some examples associated with the system, and dependent claim 11 defines an example associated with the computer-assisted method.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The disclosure will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and the accompanying drawings, in which:- FIG. 1 A is a schematic diagram of an embodiment of a system for augmenting data to estimate a state of a product;- FIG. IB is a schematic diagram illustrating the data flow between the various modules of the system according to some embodiments;- FIGS. 2 A, 2B and 2C illustrate wear and tear examples of a product in the form of a washing machine. FIG. 2A illustrates dent and rust, FIG. 2B illustrates broken glass and broken plastic components and FIG. 2C illustrates missing components; and- FIG. 3 is a flow chart of a method for augmenting data to estimate the state of the product.DETAILED DESCRIPTION

[0010] The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in theart to practice the disclosure. Other embodiments may be utilized and structural, and logical changes may be made without departing from the scope of the disclosure. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0011] Embodiments described in the context of one of the systems or methods are analogously valid for the other systems or methods.

[0012] Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0013] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0014] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0015] As used herein, the term “data” may be understood to include information in any suitable analog or digital form, for example, provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. The term data, however, is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.

[0016] As used herein, the term “module” refers to, or forms part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may include memory (shared, dedicated, or group) that stores code executed by the processor.

[0017] As used herein, the terms “first”, “second”, “third”, “fourth”, and so on, are used for purposes of clarity and do not imply order or precedence.

[0018] As used herein, the term “processor” refers to a circuit, including analog circuits, digital circuits, or hybrid circuits, or their constituent components. Any other kind of implementation of the respective functions which will be described in more detail below mayalso be understood as a “circuit” in accordance with an alternative embodiment. A digital circuit may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or a processor executing software stored in a memory, or a firmware.

[0019] As used herein, the term “obtain” refers to the processor which actively obtains the inputs, or passively receives inputs from a user interface and / or one or more sensors. The term obtain may also refer to the processor that receives or obtains inputs from a communication interface, e.g., the user interface. The processor or the dosing module may also receive or obtain the inputs via a memory, a register, and / or an analog-to-digital port.

[0020] As used herein, the term “product” includes a composite object that comprises at least one component, or a plurality of components. Non-limiting examples of a product include an electric motor (component includes one or more bolts), a washing machine, an electric drill, etc.

[0021] As used herein, the term “generative artificial intelligence model” refers to an artificial intelligence module or model capable of generating text, images, or other media data output, using one or more generative models. Generative Al models may be configured to learn the patterns and structure of their input training data, and generate / output new or novel data that has similar characteristics as the input training data.

[0022] An embodiment of the disclosure is shown in FIG. 1 A and FIG. IB, which illustrates a setup of a system 100 for augmenting data to estimate a state of a product 110. The state of the product 110 may be related to a wear and tear status or degree of damage / degradation of the product 110.

[0023] The system 100 may comprise a processor 132, the processor 132 configured to obtain a first set of image data 120 (associated with at least one image) of the product 110 at a first time T; identify, using a machine learning algorithm, a product data of the product 110 based on the first set of image data 120; retrieve a usage data of the product 110 based on the product data; assign a material type to the product 110; and predicting, using a generative artificial intelligence (Al) based model 140, a second set of image data 150 of the product 110 based on the material type, the usage data and the product data. The second set of image data 150 may be augmented data associated with the first set of image data based on a period of use of the product 110 to estimate the state of the product 110. In some embodiments, the first set of image data may correspond to a relatively new product and the second set of predicted image data may correspond to the relatively new product after being used for a pre-determined timeperiod in days, months or years. Alternatively, the first set of image data may correspond to a used product and the second set of predicted image data may correspond to a relatively new product before the product was used for a pre-determined time period in days, months or years.

[0024] The second set of image data 150 comprises an image of the product 110 predicted at a second time T+A, wherein A represents a time difference between the first time T and the second time T+A for which the product 110 has been used.

[0025] In some embodiments, the first set of image data 120 may capture an actual wear and tear condition or status of the product 110 at the first time T.

[0026] The processor 132 may comprise a product identification module 134, a usage information database 136, a component material information module 138 and a generative Al model 140. The processor 132 may be part of a data augmentation system.

[0027] In some embodiments, the processor 132 may comprise an image database 160. The image database 160 may be arranged in data communication with the processor 132 to send and / or receive at least one set of image data from the processor 132.

[0028] The system 100 may comprise an image capturing device for capturing the first set of image data 120 of the product 110. In some embodiments, the image capturing device may include an RGB image capturing device (e.g., a camera). The RGB image capturing device may be configured to capture two-dimensional (2D) images in various formats, such as .jpeg format, bitmap format, etc. In some embodiments, the image capturing device may include a 3D scanning device for point cloud data collection. In some embodiments, the image data obtained by the 3D scanning device may be converted into a computer-aided design (CAD) file.

[0029] The product identification module 134 may be configured to obtain the first set of image data 120 of the product 110 (e.g., image data of an electric motor) as input. The machine learning algorithm is then configured to estimate product data / information, such as a product name 121, a specific model 122 and / or a high-level classification of the product 110, depending on the requirements of a user, as output data of the product identification module 134. Nonlimiting examples of the machine learning algorithm may include a deep neural network (DNN), such as a convolutional neural network. In other words, the product identification module 134 may be regarded as a product classifier, the product classifier configured to identify a high-level classification of the product 110 and / or an exact model or type of the product 110. The DNN may be trained based on images that can be obtained from one or more publiclyavailable data and may be combined. Such publicly available data may be obtained, for example, from publicly available image database such as ImageNet database [1], and product specific datasets, depending on the type of products to be identified.

[0030] The usage information database 136 may be configured to receive the product name 121, the specific product model data 122 and / or the high-level classification of the product 110 as input. Based on the input, the usage information database 136 may then retrieve a usage data, such as a usage pattern 123 and a time period condition 124, which corresponds to how the product 110 may generally be used. Information from the usage information database 136 provides better understanding on the extent and type of wear and tear that the product 110 might undergo. For example, an in-the-field industrial equipment is likely to manifest different wear and tear patterns as compared to an in-house consumer appliance. The usage information database 136 may be created by collecting and curating product-specific information from, for example, product owners, domain experts, and public knowledge. In some embodiments, the time period condition 124 may be a day-to-day condition, a month-to-month condition, or a year-to-year condition.

[0031] The component material information module 138 may be configured to obtain material type 125 of the product 110 and its components. It is appreciable that depending on the type of material for each product component of the product 110, the extent and type of wear and tear may vary in nature. For example, a particular component (e.g. a door of a side-load washing machine or dryer) may be made of glass or plastic, which may be more prone to breakage / scratching compared to getting rusted. Similarly, the components (e.g. bolts, screws) made of metal such as iron may be more prone to rust, distortion in shape, etc. but may be less prone to breaking completely. In some embodiments, the material type 125 may be obtained either by (a) using electromagnetic (EM) wave spectroscopy-based methodology in combination with computer-vision based approaches or (b) by using 3D model data, such as CAD data, and bill of material analysis. In some embodiments, the material type 125 of each component of the product 110 may be stored in a material information database 180. The material information database 180 may be arranged in data communication with the processor 132 to assign the material type 125 to the product 110. The material type 125 may be processed by the component material information module 138 to obtain relevant material information 126 suitable for input into the generative Al model 140.

[0032] The generative Al model 140 may be configured to generate the second set of image data 150 based on the material type 125, the usage data (e.g., the usage pattern 123 and the day-to-day condition 124) and the product data (e.g., the product name 121 and the specific product model data 122). In some embodiments, the generative Al model 140 may comprise of a generative adversarial network (GAN) and / or a variational autoencoder (VAE).

[0033] As may be appreciated, the generative Al model 140 will have to be trained before deployment. In some embodiments, the processor 132 may comprise a training module, the training module configured to train the generative Al model 140 to generate the second set of image data 150. In some embodiment, the training module 170 is configured to train the generative Al based model 140 based on a supervised, an unsupervised, and / or a semisupervised training model, using at least one set of image data (training set and / or testing set) stored in the image database 160.

[0034] The generative Al model 140 may be trained by studying the wear and tear patterns of real-life used products, such as products marked for recycling, over a period of time. A facility in a reverse logistics operation may bring in the used products and collect the wear and tear patterns by either manually capturing images or employing computer-vision based techniques.

[0035] Wear and tear patterns in the product 110 may vary depending on where and how the product 110 is being used as well as the age of the product 110. Used products that are marked for recycling may have many types of defects due to their long-time usage. The defects present in the product 110 may be broadly categorized into two types - functional defects and non-function defects.

[0036] Functional defects may be related to whether the product 110 is in working condition or otherwise. Functional defects may not be easily identifiable by a naked eye. For example, functional defects may include spoiled components that has malfunctioned but shows no visual defects.

[0037] Non-functional defects may be related to the physical properties of the component of the product 110. Non-limiting examples include, for example, rusty and oily components, missing parts, shape deformation, dimensions, scratches etc. that can be seen by a naked eye.

[0038] In some embodiments, the generative Al model 140 may be trained using the following information: (a) a relatively large corpus of used-product images with real-life wear & tear patterns collected from, for example, reverse logistics plants and may be curated toproduce a used-product image dataset; (b) a product-wise dataset describing the typical usage patterns and conditions in which each product may be used on a day-to-day basis; and (c) material information 126 for each component of the product 110, which has been described earlier in this description. With these components, a suitable DNN architecture may be chosen and the generative Al model 140 may be iteratively trained using the above datasets until a threshold accuracy is achieved.

[0039] Once the generative Al model 140 is trained, the user may obtain the second set of image data 150 in the following manner: (a) input the first set of image data 120 of the product 110 to the DNN. The first set of image data 120 may depict the product 110 in a new condition (i.e., T = 0) or in a condition after being used for some time (i.e., T > 0); (b) identify the product 110 and obtain the product name 121 and its specific model 122 using the product identification module 134; (c) using the identified product information 121 and 122, retrieve the usage pattern 123 and / or the day-to-day condition 124 from the usage information database 136; (d) obtain the material type 125 of the product 110 and its components to retrieve the material information 126 from the material information database 180; and (e) input the above information from (b) to (d) to the generative Al model 140 to predict the second set of image data 150 with the predicted image of the product 110 after the product is being used after a further time A.

[0040] In some embodiments, the generative Al model 140 may be used for real-time wear and tear estimation of the product 110. The wear and tear estimation may provide feedback for product design. In some embodiments, the generative Al model 140 may be utilized in a decision process for recycling or repurposing of the product 110. In some embodiments, the generative Al model 140 may be utilized for a state-of-health estimation for the product 110.

[0041] FIGs. 2A to 2C show three examples of wear and tear in a used product in the form of washing machines 110A, HOB and HOC. The washing machine 110A may comprise one or more defects such as dents 202 A or rust 202B. The washing machine 110B may comprise one or more defects such as broken glass 204 A or broken plastic components 204B. The washing machine HOC may comprise one or more defects such as missing door 206A or missing control components 206B.

[0042] According to another aspect and with reference to FIG. 3, there is provided a computer-aided method 300 for augmenting data to estimate a state of a product 110, the method 300 comprising the steps of:

[0043] Step 302: obtaining a first set of image data 120 of the product 110;

[0044] Step 304: identifying, using a product identification module 134, a product data of the product 110 based on the first set of image data 120;

[0045] Step 306: retrieving a usage data of the product 110 based on the product data;

[0046] Step 308 assigning a material type 125 to the product 110; and

[0047] Step 310: predicting, using a generative artificial intelligence (Al) based model 140, a second set of image data 150 of the product 110 based on the material type 125, the usage data and the product data. The second set of image data 150 may be augmented data associated with the first set of image data 120 based on a period of use of the product 110 to estimate the state of the product 110. In some embodiments, the period of use comprises a first time T associated with the first set of image data (120), and a second time T+A associated with the predicted second set of image data (150), wherein A represents a time difference between the first time T and the second time T+A for which the product (110) has been used.

[0048] According to another aspect of the disclosure, there is a computer program, the computer program comprising instructions to execute the computer-assisted method 300. In some embodiments, there may comprise a non-transitory computer readable medium configured to store executable software instructions thereon, such that when executed, performs the method 300.Reference[1] Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. (* = equal contribution) ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015

Claims

CLAIMS1. A system (100) for augmenting data to estimate a state of a product (110), the system (100) comprising a processor (132), the processor (132) being configured to: obtain a first set of image data (120) of the product (110); identify, using a product identification module (134), a product data (121, 122) of the product (110) based on the first set of image data (120); retrieve a usage data (123, 124) of the product (110) based on the product data (121, 122); assign a material type (125) to the product (110); and predict, using a generative artificial intelligence (Al) based model (140), a second set of image data (150) of the product (110) based on the material type (125), the usage data (123, 124) and the product data (121, 122); wherein the second set of image data (150) is augmented data associated with the first set of image data (120) based on a period of use of the product (110) to estimate the state of the product (110).

2. The system (100) of claim 1, wherein the period of use comprises a first time T associated with the first set of image data (120), and a second time T+A associated with the predicted second set of image data (150), wherein A represents a time difference between the first time T and the second time T+A for which the product (110) has been used.

3. The system (100) of claim 1 or 2, further comprising a training module (170), the training module (170) configured to train the generative Al based model (140) to generate the second set of image data (150).

4. The system (100) of any one of the preceding claims, further comprising an image database (160), the image database (160) arranged in data communication with the processor (132) to send and / or receive at least one set of image data (120, 150) from the processor (132).

5. The system (100) of claim 4, wherein the training module (170) is configured to train the generative Al based model (140) using the at least one set of image data (120, 150) stored in the image database (160).

6. The system (100) of any one of the preceding claims, wherein the product identification module (134) comprises a machine learning algorithm, and wherein the machine learning algorithm comprises a deep neural network (DNN).

7. The system (100) of any one of the preceding claims, wherein the product (110) comprises a plurality of components, and the processor (132) is configured to generate the second set of image data (150), especially of each of the plurality of components.

8. The system (100) of any one of the preceding claims, further comprising a material information database (180), the material information database (180) arranged in data communication with the processor (132) to assign the material type (125) to the product (110).

9. The system (100) of any one of the preceding claims, wherein the estimated state of the product (110) is indicative of a wear and tear of the product (110).

10. A computer-aided method (300) for augmenting data to estimate a state of a product (110), the method (300) comprising the steps of: obtaining (302) a first set of image data (120) of the product (110); identifying (304), using a product identification module (134), a product data (121, 122) of the product (110) based on the first set of image data (120); retrieving (306) a usage data (123, 124) of the product (110) based on the product data (121, 122); assigning (308) a material type (125) to the product (110); and predicting (310), using a generative artificial intelligence (Al) based model (140), a second set of image data (150) of the product (110) based on the material type (125), the usage data (123, 124) and the product data (121, 122);wherein the second set of image data (150) is augmented data associated with the first set of image data (120) based on a period of use of the product (110) to estimate the state of the product (110).

11. The method (300) of claim 10, further comprising a step of training the generative Al based model (140) to generate the second set of image data (150).

12. A computer program, the computer program comprising instructions to execute the computer-aided method (300) according to claim 10 or 11.

Images