Method and system for generating datasets for training a dnn

Abstract

A method and system for generating a point cloud dataset for training a Deep Neural Network, DNN. The method comprises the steps of a) providing a first CAD data file representative of an object; b) removing data representative of at least one component of the object from the first CAD datafile to generate a modified CAD data file; and c) converting the modified CAD data file to a modified point cloud data of the object for the point cloud dataset.

Description

[0001] METHOD AND SYSTEM FOR GENERATING DATASETS FOR TRAINING A DNN

[0002] FIELD OF INVENTION

[0003] The present invention relates broadly to method and system for generating a point cloud dataset for training a DNN, in particular to a method and system for CAD driven 3D data augmentation.

[0004] BACKGROUND

[0005] Any mention and / or discussion of prior art throughout the specification should not be considered, in any way, as an admission that this prior art is well known or forms part of common general knowledge in the field.

[0006] In industry 4.0, Deep Neural Networks (DNNs), when combined with industrial internet-of- things (IIoT), have paved the path for applications like automatic predictive maintenance, automatic defect identification, spare-part identification, and so on.

[0007] The accuracy of such a DNN backed system mainly depends on following factors:

[0008] 1. Specific DNN architecture being used

[0009] 2. Quality of dataset for training the dataset

[0010] DNN architectures have evolved drastically over the last couple of years e.g., while initial models like Fast RCNN were two stage models, the newer ones are single stage. Also, the number of hidden layers has been increasing lately with earlier models like YoLovl having 26+ layers whereas, the later models like YoLov5 has 87+ layers. While there are many publicly available DNNs e.g., YoLo [7], SSD [8], PointNet [9] etc., it is important to train these models on a high-quality and large-enough dataset. Gathering (or generating) a large amount of high-quality dataset can be a laborious task, especially for use-cases where no public dataset is available e.g., specific automobile spare-part classification task.

[0011] One of the industry-wide techniques to effectively increase the dataset size is data augmentation. Some examples of such techniques include random cropping, flipping, rotating images, adding random noise, random content erasing / crop, color space manipulation and so on. Although these techniques are quite popular, they seem to create “artificial” data. For example, a randomly cropped 3-dimensional (3D) point cloud, from an original product’s point cloud, might not even represent a real-life product anymore. A random crop operation might remove parts of the product in a manner that is not likely to happen in real-life.

[0012] Figures 1A-C show some examples of data augmentation techniques using random crop applied to eAxle (electronic axle). For this type of augmentation, one randomly crops a subset of the 3D data within a specified region or bounding box. Random crop enables to focus on specific regions of interest. The crop region can be randomly generated or predefined. As can be seen from the samples of cropped 3D data images in Figures 1A-C, the random crop has arbitrarily removed certain areas of the eAxle - resulting in an “artificial” 3D point cloud of eAxle.

[0013] As another example, a random noise data augmentation applied to eAxle adds gaussian (or any other) noise to the specified areas of eAxle. Adding random noise to a 3D point cloud can introduce variations and imperfections. Noise can simulate sensor noise or measurement inaccuracies. Gaussian or uniform noise can be added to the coordinates or voxel values. Figures 2A-B show examples of data augmentation techniques for adding random noise applied to eAxle.

[0014] Many data augmentation techniques, targeting deep learning system, exist in literature. The techniques proposed in [1] focus on data augmentation for deep learning models specifically targeting medical domain images. Authors in [2] & [3] have studied the use of traditional as well as state-of-the-art machine learning based techniques for data augmentation.

[0015] Although these techniques are quite popular, they seem to create “artificial” data. For example, a randomly cropped 3D point cloud, from an original product’s point cloud, might not even represent a real-life product anymore. A random crop operation might remove parts of the product in a manner that is not likely to happen in real-life.

[0016] Embodiments of the present invention seek to address at least one of the above problems.

[0017] SUMMARY

[0018] In accordance with a first aspect of the present invention, there is provided a method for generating a point cloud dataset for training a Deep Neural Network, DNN, comprising the steps of: a) providing a first CAD data file representative of an object; b) removing data representative of at least one component of the object from the first CAD datafile to generate a modified CAD data file; and c) converting the modified CAD data file to a modified point cloud data of the object for the point cloud dataset.

[0019] In accordance with a second aspect of the present invention, there is provided a system for generating a point cloud dataset for training a DNN, the system comprising a processing unit configured to a) provide a first CAD data file representative of an object; b) remove data representative of at least one component of the object from the first CAD datafile to generate a modified CAD data file; and c) convert the modified CAD data file to a modified point cloud data of the object for the point cloud dataset.

[0020] In accordance with a third aspect of the present invention, there is provided a Deep Neural Network having been trained by a dataset generated by a method according to the first aspect or a system according to the second aspect.

[0021] In accordance with a fourth aspect of the present invention, there is provided a use of the Deep Neural Network according to the third aspect for image classification, automatic predictive maintenance, automatic defect identification, or spare-part identification.

[0022] BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments of the invention will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:

[0024] Figures 1 A shows an example of data augmentation techniques using random crop applied to eAxle.

[0025] Figures IB shows an example of data augmentation techniques using random crop applied to eAxle.

[0026] Figures 1C shows an example of data augmentation techniques using random crop applied to eAxle.

[0027] Figures 2A shows an example of data augmentation techniques for adding random noise applied to eAxle.

[0028] Figures 2B shows an example of data augmentation techniques for adding random noise applied to eAxle.

[0029] Figures 3A shows the resulting 3D point cloud from the CAD file for eAxle, specifically the original or complete eAxle.

[0030] Figures 3B shows the resulting 3D point cloud with the central motor removed from the CAD file for eAxle, according to an example embodiment.

[0031] Figures 3C shows the resulting 3D point cloud with the side panel removed from the CAD file for eAxle, according to an example embodiment.

[0032] Figures 3D shows the resulting 3D point cloud with the side gears removed from the CAD file for eAxle, according to an example embodiment.

[0033] Figure 4 shows a flow-chart illustrating a method for generating a point cloud dataset for DNN architectures, according to an example embodiment. Figure 5 shows a schematic drawing illustrating a system for generating a point cloud dataset for DNN architectures, according to an example embodiment.

[0034] DETAILED DESCRIPTION

[0035] Embodiments of the present invention utilize an opportunity to leverage the Computer-Aided Design (CAD) model of a product to create additional and “real” datasets, for example, but not limited to, 3D data. Example embodiments use CAD model of a product and remove its components, one by one, while converting each resulting CAD model to a 3D point cloud. This process is repeated multiple times and results in a database of “realistic” 3D point clouds suitable for used, end-of-life products, according to example embodiments.

[0036] For a given object, e.g. a product P and its CAD data file C, an example embodiment uses the following algorithm to create additional realistic 3D data:

[0037] 1. Export C as a point cloud (PCk ) convert (Q — PCk where k = 0

[0038] 2. For k G {1,2, 3.... I } repeat the following steps: a. Select and remove a component c from the CAD file C, where c G COMc (list of all the components in Q. The remaining CAD file now can be represented by Ck b. Export resulting CAD file (C*) as point cloud PCk) convert Ck) —> PCk c. Add resulting point cloud to the global list of point clouds:

[0039] PCiist [k] = PCk

[0040] That is, in an example embodiment the “actual product parts” are iteratively removed, one at a time, from a CAD file and the resulting CAD file is converted to the corresponding point cloud data. This process is repeated multiple times, therefore advantageously providing “realistic” 3D point cloud data that is visually and geometrically closer to the state of an end-of-life product. It is noted that in different embodiments, in each iteration one or more parts of the product may be removed.

[0041] Figures 3 A-D show the resulting 3D point clouds while removing components one by one from the CAD file for eAxle, according to a non-limiting example embodiment. It can be seen that the 3D data looks more realistic and, unlike e.g. existing random crop and random noise techniques, CAD based data generation according to an example embodiment results in more accurate point clouds. Embodiments of the present invention can have the following features and associated advantages:

[0042] 1. Highly accurate missing component dataset:

[0043] The point clouds obtained from according to an example embodiment represent the 3D data of a product when some of its components are missing. This dataset can be extremely useful for missing component detection use-case in a used product’s quality inspection. Missing component problem is particularly important for quality inspection as well as automated disassembly of used & end-of-life products.

[0044] 2. Large amount of realistic 3D point cloud data:

[0045] Unlike existing data augmentation techniques e.g., random crop, random noise etc., the dataset created according to an example embodiment is real and represents an actual product. In terms of training a machine learning algorithm, such a dataset advantageously adds much more diversity and value to the training process compared to the dataset obtained using existing data augmentation techniques.

[0046] Figure 4 shows a flow-chart 400 illustrating a method for generating a point cloud dataset for training a Deep Neural Network, DNN, according to an example embodiment. At step 402, a first CAD data file representative of an object is provided. At step 404, data representative of at least one component of the object is removed from the first CAD datafile to generate a modified CAD data file. At step 406, the modified CAD data file is converted to a modified point cloud data of the object for the point cloud dataset.

[0047] The method may comprise converting the first CAD data file to an unmodified point cloud data of the object for the point cloud dataset.

[0048] The method may comprise performing steps 404 and 406 iteratively to remove at least one component of the object in each iteration for generating a plurality of respective modified point cloud data of the object for the point cloud dataset.

[0049] The method may comprise adding the point cloud dataset to a global list of point clouds for the object.

[0050] The method may comprise training the DNN based on at least the point cloud dataset.

[0051] Figure 5 shows a schematic drawing illustrating a system 500 for generating a point cloud dataset 510 for training a DNN 503, according to an example embodiment. The system 500 comprises a processing unit (501) configured to a) provide a first CAD data file 502 representative of an object; b) remove data representative of at least one component of the object from the first CAD datafile 502 to generate a modified CAD data file 506; and c) convert the modified CAD data file 506 to a modified point cloud data 508 of the object for the point cloud dataset 510.

[0052] The processing unit 501 may be configured to convert the first CAD data file 502 to an unmodified point cloud data 504 of the object for the point cloud dataset 510.

[0053] The processing unit 501 may be configured to perform steps b) and c) iteratively to remove at least one component of the object in each iteration for generating a plurality of respective modified point cloud data of the object for the point cloud dataset 510. The processing unit 501 may be configured to add the point cloud dataset 510 to a global list of point clouds for the object.

[0054] The system may be configured to be coupled to the DNN 503 for training of the DNN 503 based on at least the point cloud dataset 510.

[0055] In one embodiment, a Deep Neural Network having been trained by a dataset generated by a method according to embodiments of the present invention or a system according to embodiments of the present invention is provided.

[0056] In one embodiment, use of the Deep Neural Network according to to example embodiments for image classification, automatic predictive maintenance, automatic defect identification, or spare-part identification is provided.

[0057] Aspects of the systems and methods described herein, such as system 400, processing unit 401 for the use of CAD model of a product and remove its respective components, while converting each resulting CAD model to a 3D point cloud, and / or DNN 503 may be implemented on computing device(s), including cloud-based computing device(s) and / or Internet-of-Things computing device(s), for example as functionality programmed into any of a variety of circuitry, including programmable logic devices (PLDs), such as field programmable gate arrays (FPGAs), programmable array logic (PAL) devices, electrically programmable logic and memory devices and standard cell-based devices, as well as application specific integrated circuits (ASICs). Some other possibilities for implementing aspects of the system include: microcontrollers with memory (such as electronically erasable programmable read only memory (EEPROM)), embedded microprocessors, firmware, software, etc. Furthermore, aspects of the system may be embodied in microprocessors having software-based circuit emulation, discrete logic (sequential and combinatorial), custom devices, fuzzy (neural) logic, quantum devices, and hybrids of any of the above device types. Of course the underlying device technologies may be provided in a variety of component types, e.g., metal-oxide semiconductor field-effect transistor (MOSFET) technologies like complementary metal-oxide semiconductor (CMOS), bipolar technologies like emitter-coupled logic (ECL), polymer technologies (e.g., silicon-conjugated polymer and metal-conjugated polymer-metal structures), mixed analog and digital, etc. The various functions or processes disclosed herein may be described as data and / or instructions embodied in various computer-readable media, in terms of their behavioral, register transfer, logic component, transistor, layout geometries, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, non-volatile storage media in various forms (e.g., optical, magnetic or semiconductor storage media) and carrier waves that may be used to transfer such formatted data and / or instructions through wireless, optical, or wired signaling media or any combination thereof. When received into any of a variety of circuitry (e.g. a computer), such data and / or instruction may be processed by a processing entity (e.g., one or more processors).

[0058] It will be appreciated by a person skilled in the art that numerous variations and / or modifications may be made to the present invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive. Also, the invention includes any combination of features described for different embodiments, including in the summary section, even if the feature or combination of features is not explicitly specified in the claims or the detailed description of the present embodiments.

[0059] In general, in the following claims, the terms used should not be construed to limit the systems and methods to the specific embodiments disclosed in the specification and the claims, but should be construed to include all processing systems that operate under the claims. Accordingly, the systems and methods are not limited by the disclosure, but instead the scope of the systems and methods is to be determined entirely by the claims.

[0060] Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise," "comprising," and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in a sense of "including, but not limited to." Words using the singular or plural number also include the plural or singular number respectively. Additionally, the words "herein," "hereunder," "above," "below," and words of similar import refer to this application as a whole and not to any particular portions of this application. When the word "or" is used in reference to a list of two or more items, that word covers all of the following interpretations of the word: any of the items in the list, all of the items in the list and any combination of the items in the list.

[0061] References:

[0062] 1. Chlap, Phillip, et al. "A review of medical image data augmentation techniques for deep learning applications." Journal of Medical Imaging and Radiation Oncology 65.5 (2021): 545- 563.

[0063] 2. Wang, Jason, and Luis Perez. "The effectiveness of data augmentation in image classification using deep learning." Convolutional Neural Networks Vis. Recognit 11.2017 (2017): 1-8. 3. Shorten, C., Khoshgoftaar, T.M. A survey on Image Data Augmentation for Deep Learning.

[0064] J Big Data 6, 60 (2019). https: / / doi.org / 10.1186 / s40537-019-0197-0

[0065] 4. IBM Maximo System for spare-part-classification https: / / www.ibm.com / blog / fmd-the- right-spare-part-with-ai /

[0066] 5. Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. (* = equal contribution) ImageNet Large Scale Visual Recognition Challenge. IJCV, 2015

[0067] 6. Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde- Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua https: / / proceedings.neurips.cc / paper_files / paper / 2014 / file / 5ca3e9bl22f61f8f06494c97blafccf 3 -Paper, pdf

[0068] 7. Joseph Redmon, Ali Farhadi. YOL09000: Better, Faster, Stronger. http: / / arxiv.org / abs / 1612.08242

[0069] 8. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed, Cheng- Yang Fu, Alexander C. Berg: SSD: Single Shot MultiBox Detector. https: / / arxiv.org / abs / 1512.02325

[0070] 9. Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, https: / / arxiv.org / abs / 1612.00593

[0071] 10. Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, Vladlen Koltun: Point Transformer: https: / / openaccess.thecvf.com / content / ICCV2021 / papers / Zhao_Point_Transformer_ICCV_20 21_paper.pdf

Claims

CLAIMS1. A method for generating a point cloud dataset (510) for training a Deep Neural Network, DNN, (503) comprising the steps of: a) providing a first CAD data file (502) representative of an object; b) removing data representative of at least one component of the object from the first CAD datafile (502) to generate a modified CAD data file (506); and c) converting the modified CAD data file (506) to a modified point cloud data (508) of the object for the point cloud dataset (510).

2. The method of claim 1, comprising converting the first CAD data file (502) to an unmodified point cloud data (504) of the object for the point cloud dataset (510).

3. The method of claims 1 or 2, comprising performing steps b) and c) iteratively to remove at least one component of the object in each iteration for generating a plurality of respective modified point cloud data of the object for the point cloud dataset (510).

4. The method of any one of claims 1 to 3, comprising adding the point cloud dataset to a global list of point clouds for the object.

5. The method of any one of claims 1 to 4, comprising training the DNN (503) based on at least the point cloud dataset (510).

6. A system (500) for generating a point cloud dataset (510) for training a DNN (503), the system comprising a processing unit (501) configured to a) provide a first CAD data file (502) representative of an object; b) remove data representative of at least one component of the object from the first CAD datafile (502) to generate a modified CAD data file (506); and c) convert the modified CAD data file (506) to a modified point cloud data (508) of the object for the point cloud dataset (510).

7. The system of claim 6, wherein the processing unit (501) is configured to convert the first CAD data file (502) to an unmodified point cloud data (504) of the object for the point cloud dataset (510).

8. The system of claims 6 or 7, wherein the processing unit (501) is configured to perform steps b) and c) iteratively to remove at least one component of the object in each iteration for generating a plurality of respective modified point cloud data of the object for the point cloud dataset (510).

9. The system of any one of claims 6 to 8, wherein the processing unit (501) is configured to add the point cloud dataset to a global list of point clouds for the object.

10. The system of any one of claims 6 to 9, configured to be coupled to the DNN (503) for training of the DNN (503) based on at least the point cloud dataset (510).

11. Deep Neural Network having been trained by a dataset generated by a method according to claims 1 to 5 or a system according to claims 6 to 10.

12. Use of the Deep Neural Network according to claim 11 for image classification, automatic predictive maintenance, automatic defect identification, or spare-part identification.

Images