Experiential learning in a virtual world

Abstract

To provide an improved method, system, and program for machine learning with a dataset of scenes including spatially reconfigurable objects.SOLUTION: A method provided herein comprises: obtaining a dataset of virtual scenes belonging to a first domain; obtaining a test dataset of real scenes belonging to a second domain; determining a third domain that is closer to the second domain than the first domain in terms of data distribution; and learning a domain-adaptive neural network based on the third domain, the domain-adaptive neural network being configured for inference of spatially reconfigurable objects in a real scene.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems, and programs for machine learning using datasets of scenes containing spatially reconstructable objects. [Background technology]

[0002] For the design, engineering and manufacturing of objects, numerous systems and programs are offered on the market. CAD is an acronym for Computer-Aided Design, e.g., related to software solutions for designing objects. CAE is an acronym for Computer-Aided Engineering, e.g., related to software solutions for simulating the physical behavior of a future product. CAM is an acronym for Computer-Aided Manufacturing, e.g., related to software solutions for defining manufacturing processes and operations. In such Computer-Aided Design systems, the graphical user interface plays a key role regarding the efficiency of the technique. These technologies may be incorporated within Product Lifecycle Management (PLM) systems. PLM refers to a business strategy that helps companies share product data, apply common processes and leverage corporate knowledge to help develop products from concept to life, beyond long-term corporate concepts. The PLM solutions offered by Dassault Systèmes (under the trademarks CATIA, ENOVIA and DELMIA) provide an Engineering Hub that organizes product engineering knowledge, a Manufacturing Hub that manages manufacturing engineering knowledge and a Corporate Hub that allows companies to connect to both the Engineering and Manufacturing Hubs. Overall, the system provides an open object model that links products, processes, and resources to enable dynamic knowledge-based product creation and decision support that drives optimized product definition, manufacturing preparation, production, and service.

[0003] In this and other contexts, machine learning is gaining widespread importance.

[0004] The following papers are relevant to this field and are referenced in this paper: [1] A. Gaidon, Q. Wang, Y. Cabon, and E. Vig, “Virtual worlds as proxy for multi-object tracking analysis,” in CVPR, 2016. [2] SR Richter, V. Vineet, S. Roth, and V. Koltun, “Playing for data: Ground truth from computer games,” in ECCV, 2016. [3] M. Johnson-Roberson, C. Barto, R. Mehta, SN Sridhar, K. Rosaen, and R. Vasudevan, “Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?” in ICRA, 2017. [4] SR Richter, Z. Hader, and V. Koltun, “Playing for benchmarks,” in ICCV, 2017. [5] S. Hinterstoisser, V. Lepetit, P. Wohlhart, and K. Konolige, “On pretrained image features and synthetic images for deep learning,” in arXiv:1710.10710, 2017. [6] D. Dwibedi, I. Misra, and M. Hebert, “Cut, paste and learn: Surprisingly easy synthesis for instance detection,” in ICCV, 2017. [7] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), 2017. [8] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” in CVPR Workshop on Autonomous Driving (WAD), 2018. [9] K. He, G. Gkioxari, P. Dollar, and R. Girshick. “Mask r-cnn“. arXiv:1703.06870, 2017.

[10] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2017, 2017. URL http: / / www.image-net.org / challenges / LSVRC / 2017 / .

[11] Y. Chen, W. Li, and L. Van Gool. ROAD: Reality oriented adaptation for semantic segmentation of urban scenes. In CVPR, 2018.

[12] Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool. Domain adaptive Faster R-CNN for object detection in the wild. In CVPR, 2018.

[13] Y. Zou, Z. Yu, B. Vijaya Kumar, and J. Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV, September 2018.

[14] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? The KITTI vision benchmark suite,” in CVPR, 2012.

[15] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield. Structured domain randomization: Bridging the reality gap by contextaware synthetic data. arXiv preprint arXiv:1810.10093, 2018.

[16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.

[17] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. Microsoft COCO: Common objects in ´ context. In ECCV. 2014.

[18] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR, abs / 1506.01497, 2015.

[19] D. E. Rumelhart , G. E. Hinton , R. J. Williams, Learning internal representations by error propagation, Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, MIT Press, Cambridge, MA, 1986.

[20] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. CoRR, abs / 1703.10593, 2017.

[21] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NIPS), 2017.

[22] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.

[0005] Deep learning techniques have shown remarkable performance in several areas, such as 2D / 3D object recognition and detection, semantic segmentation, pose estimation, or motion capture (see [9, 10]). However, such techniques usually (1) require huge amounts of training data to perform at their full potential, and (2) often require expensive manual labeling (also known as annotation).

[0006] (1) Collecting hundreds of thousands of labeled data can be very challenging in practice in some domains, such as manufacturing, health, or indoor motion capture, due to data scarcity or confidentiality reasons. Publicly available datasets usually contain objects from everyday life that have been manually labeled via crowdsourcing platforms (see [16, 17]).

[0007] (2) In addition to the data collection problem, labeling is very time-consuming. The annotation of the well-known ImageNet dataset (see

[16] ), where each image is labeled by a "unique" single label, has been going on for years. Pixel-by-pixel labeling requires on average more than one hour per image. Also, manual annotation can present some errors and lack of accuracy. Some annotations require domain-specific know-how (e.g., labeling manufacturing equipment).

[0008] For all these reasons, the use of virtual data to train deep models has gained attention in recent years (see [1, 2, 3, 4, 5, 6, 7, 8, 15]). Virtual (also called synthetic) data is computer-generated data that is not generated by real events but is created using CAD or 3D modeling software. Indeed, virtual data can be generated to meet very specific needs or conditions that are not available with existing (real) data. This is useful when privacy requires limiting the availability or use of data or when the necessary data for a test environment simply does not exist. It should be noted that with virtual data, labeling is both cost-free and error-free.

[0009] However, the inherent domain difference between virtual and real data (also known as the reality gap) can prevent the learned models from being competent for real-world scenarios. This is mainly due to the challenge of overfitting, where the network learns details that are only present in the virtual data and is unable to generalize well enough to extract useful representations of the real data.

[0010] One method that has gained popularity recently, especially for perception problems involving autonomous vehicles, is to collect photorealistic virtual data, such as video game data, that can be automatically annotated at low cost.

[0011] For example, Richter et al. (see [2]) constructed a large-scale synthetic urban scene dataset for semantic segmentation from the GTAV game. The GTA-based data (see [2,3,4]) uses a large number of assets to generate a variety of realistic driving environments that can be automatically labeled at the pixel level (~25K images). Such photorealistic virtual data often does not exist for domains other than driving and urban situations. For example, existing virtual manufacturing environments are non-photorealistic CAD-based environments (no video games exist for the manufacturing domain). Photorealism is not required in manufacturing specifications. Moreover, having shading and lighting can be confusing in such virtual environments. Therefore, all efforts are focused on making semantically and geometrically accurate CAD models rather than photorealistic CAD models. Moreover, due to the high variability between different factories and factory layouts, it is not appropriate to make a photorealistic manufacturing environment for training purposes. Unlike urban contexts, where urban scenes have strong specificities (e.g., size and spatial relationships of buildings, streets, cars, etc.), there is no “intrinsic” spatial structure for manufacturing scenes. We also note that some previous works, such as VKITTI (see [1]), generated replicas of real-world driving scenes and were therefore highly correlated and lacking variability with their original real-world data.

[0012] Moreover, even with a high level of realism, it was not clear how to effectively use photorealistic data to train neural networks to act on real data. Often, cross-domain adaptation components (see [11, 12, 13]) are used while training neural networks to reduce the gap between the virtual and real representations.

[0013] In summary, photorealistic virtual datasets require either a carefully designed simulation environment or the presence of annotated real data as a starting point, which is not feasible in many fields such as manufacturing.

[0014] To mitigate such difficulties, Tobin et al. (see [7]) introduced the concept of domain randomization (DR), where realistic rendering is avoided in favor of random variation. Their approach randomly varies the texture and color of foreground objects, the background image, the number of lights in the scene, the pose of the lights, the camera position, and the foreground objects. The goal is to close the reality gap by generating virtual data with enough variation that the network sees the real-world data as simply another variation. Using DR, they trained a neural network to estimate the 3D world positions of various shape-based objects relative to a robot arm fixed to a table. Recent work (see [8]) demonstrated the ability of DR to achieve the state of the art in 2D bounding box detection of automobiles in the real-world KITTI dataset (see

[14] ). However, DR requires very large amounts of data to train (~100K per object class) given the amount of variation, and in many cases networks find it difficult to learn the correct features, and the lack of context prevents DR from detecting small or occluded objects. Indeed, the results of that work were limited to: (1) large vehicles that were fully visible (KITTI Easy dataset); and (2) when used for complex inference tasks such as object detection, the approach requires enough pixels within the bounding box for the learning model to make a decision without the surrounding context. All of these shortcomings were quantitatively confirmed (see

[15] ).

[0015] To handle more challenging criteria (e.g., smaller vehicles, partial occlusions), the authors in

[15] proposed a variant of DR where they exploit urban contextual information and urban structural scenes by randomly placing objects according to a probability distribution arising from the specific problem at hand (i.e., car detection in an urban context) instead of the uniform probability distribution used in previous DR-based works.

[0016] Also, DR-based methods (see [5, 6, 7, 15]) mainly focus on “easy” object classes (e.g., the “car” class) with little intra-class variability. Significant intra-class variability can cause significant performance degradation of these techniques, as the amount of variation that has to be considered during data creation can grow exponentially. This is the case for spatially reconstructible objects, e.g., articulated objects.

[0017] Spatially reconfigurable objects, or articulated objects, are objects that have components attached via joints and can move relative to one another giving each object type an infinite range of possible statuses.

[0018] The same problem can arise while dealing with multiple object classes (inter-class and intra-class variability).

[0019] Within this context, there remains a need for improved machine learning methods using datasets of scenes containing spatially reconstructable objects. DISCLOSURE OF THEINVENTION

[0020] Accordingly, the present invention provides a computer-implemented method of machine learning. The method includes providing a dataset of a virtual scene. The dataset of the virtual scene belongs to a first domain. The method further includes providing a test dataset of a real scene. The test dataset belongs to a second domain. The method further includes determining a third domain. The third domain is closer to the second domain than the first domain in terms of data distribution. The method further includes training a domain-adaptive neural network based on the third domain. The domain-adaptive neural network is a neural network configured for inference of spatially reconfigurable objects in real scenes.

[0021] The method may include one or more of the following. The intermediate domain determination involves, for each scene in the test dataset: Transforming the scenes from the test dataset into another scene that is closer to the training domain in terms of data distribution. Scene conversion includes: A virtual scene is generated from a scene in the test dataset. Another scene is inferred based on the scene in the test dataset and the generated virtual scene. The step of generating virtual scenes from the scenes of the test dataset includes applying a virtual scene generator to the scenes of the test dataset, the virtual scene generator being a machine learning neural network configured to infer virtual scenes from scenes of the test domain. The virtual scene generator is trained on a dataset of scenes, each of which contains one or more spatially reconstructible objects. The step of determining the intermediate domain further comprises a step of blending a scene of the test dataset with the generated virtual scene, the blending resulting in another scene. The mixture of scenes from the test dataset and the generated virtual scene is a linear mixture. · The test dataset consists of real scenes. Each real scene in the test dataset is a real manufacturing scene containing one or more spatially reconfigurable manufacturing tools, and a domain-adaptive neural network is configured for inference on the spatially reconfigurable manufacturing tools in the real manufacturing scenes. The training domain includes a training dataset of virtual scenes, each of which contains one or more spatially reconfigurable objects. Each virtual scene of the dataset of virtual scenes is a virtual manufacturing scene comprising one or more spatially reconfigurable manufacturing tools, and the domain adaptive neural network is configured to infer the spatially reconfigurable manufacturing tools in the real manufacturing scene, and / or the data obtained from the training domain includes a scene of another intermediate domain, and the domain adaptive neural network has been trained on the another intermediate domain, the another intermediate domain being closer to the intermediate domain than the training domain in terms of data distribution.

[0022] Furthermore, a domain-adaptive neural network that can be trained according to this method is proposed.

[0023] A computer program comprising instructions for carrying out the method is further provided.

[0024] Further provided is an apparatus comprising a domain adaptive neural network and / or a data storage medium having a computer program recorded thereon.

[0025] The apparatus may form or function as a non-transitory computer-readable medium, such as on a SaaS or other server or cloud-based platform. The device may alternatively comprise a processor coupled to the data storage medium. Thus, the apparatus may form in whole or in part a computer system (e.g., the apparatus is a subsystem of the overall system). The system may further comprise a graphical user interface coupled to the processor. [Brief description of the drawings]

[0026] [Figure 1] 1 shows a flowchart of a computer-implemented process integrating an example method. [Diagram 2] 1 is a flowchart illustrating an example computer-implemented process and / or method. [Diagram 3] 1 is a flowchart illustrating an example computer-implemented process and / or method. [Figure 4] 1 is a flowchart illustrating an example computer-implemented process and / or method. [Diagram 5] 1 shows an example of a graphical user interface of the system. [Figure 6] An example of a system is shown. [Figure 7] Processes and / or methods are presented. [Figure 8] Processes and / or methods are presented. [Figure 9] Processes and / or methods are presented. [Figure 10] Processes and / or methods are presented. [Figure 11] Processes and / or methods are presented. [Figure 12] Processes and / or methods are presented. [Figure 13] Processes and / or methods are presented. [Figure 14] Processes and / or methods are presented. [Figure 15] Processes and / or methods are presented. [Figure 16] Processes and / or methods are presented. [Figure 17] Processes and / or methods are presented. [Figure 18] Processes and / or methods are presented. [Figure 19] Processes and / or methods are presented. [Figure 20] Processes and / or methods are presented. [Figure 21]Processes and / or methods are presented. [Figure 22] Processes and / or methods are presented. [Figure 23] Processes and / or methods are presented. [Figure 24] Processes and / or methods are presented. [Diagram 25] Processes and / or methods are presented. [Figure 26] Processes and / or methods are presented. [Figure 27] Processes and / or methods are presented. [Figure 28] Processes and / or methods are presented. [Figure 29] Processes and / or methods are presented. [Diagram 30] Processes and / or methods are presented. [Diagram 31] Processes and / or methods are presented. [Diagram 32] Processes and / or methods are presented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The present invention provides a computer-implemented method for machine learning using a dataset of scenes containing spatially reconstructable objects.

[0028] In particular, the present invention provides a first method of machine learning. The first method includes providing a dataset of a virtual scene. The dataset of the virtual scene belongs to a first domain. The first method further includes providing a test dataset of a real scene. The test dataset belongs to a second domain. The first method further includes determining a third domain. The third domain is closer to the second domain than the first domain in terms of data distribution. The first method further includes training a domain adaptation neural network based on the third domain. The domain adaptation neural network is a neural network configured for inference of spatially reconfigurable objects in real scenes. Hereinafter, this first method may be referred to as a "domain adaptation learning method."

[0029] Domain adaptation learning methods form an improved method of machine learning using datasets of scenes containing spatially reconstructable objects.

[0030] In particular, the domain adaptation learning method provides a domain adaptation neural network that can infer spatially reconfigurable objects in real scenes, which allows the domain adaptation learning method to be used in the inference process of spatially reconfigurable objects in real manufacturing scenes, for example, in the inference process of spatially reconfigurable manufacturing tools (e.g., industrial articulated robots).

[0031] Moreover, the domain-adapted neural network cannot directly train on a dataset of a real scene, but the domain-adapted neural network can infer real spatially reconfigurable objects in a real scene. Such an improvement is particularly achieved by determining a third domain according to a domain adaptation learning method. In fact, the first domain may be relatively far from the second domain (i.e., the domain of the test dataset of the domain-adapted neural network, which is made on a real scene) in terms of data distribution, since it is made on a virtual scene. Such a reality gap may make training of a domain-adapted neural network directly based on a dataset of a virtual scene difficult and / or inaccurate. Such training may actually result in a domain-adapted neural network that may generate inaccurate results in terms of inferring spatially reconfigurable objects in a real scene. Instead, the domain adaptation learning method is based on determining a third domain that is closer to the second domain than the first domain in terms of data distribution, and training of the domain-adapted neural network on this third domain. This allows training of a domain-adaptive neural network that can accurately infer spatially reconstructable objects in a real scene while the only dataset provided (i.e., given as input to the domain adaptation training method) is in fact a dataset of a virtual scene, in other words, both the training and the trained domain-adaptive neural network are more robust.

[0032] This means that the dataset of virtual scenes is theoretically intended to be the training set (or learning set) of the domain adaptation neural network, and that the domain adaptation learning method includes a stage of pre-processing of the training set, i.e., the determination of the third domain, before the actual learning, so as to train the domain adaptation neural network with the aforementioned improvements. In other words, the true training / learning domain of the domain adaptation neural network is the third domain, originating from the pre-processing of the domain originally intended to be the training / learning domain (i.e., the first domain) of the domain adaptation neural network. Nevertheless, relying on the dataset of virtual scenes as the theoretical training set (i.e., processed before being used for training, as described above) makes it possible to use datasets with a large number of scenes. Indeed, in general, there are no, or at least very few, available large datasets of real scenes containing spatially reconfigurable objects, such as manufacturing scenes containing spatially reconfigurable manufacturing tools. Such datasets are difficult to obtain in practice due to privacy / confidentiality issues, e.g., manufacturing scenes with spatially reconfigurable manufacturing tools such as robots are typically confidential and therefore not publicly available. Moreover, even if such a large dataset of real scenes is available, it may contain many annotation (e.g., labeling) errors, and annotating spatially reconfigurable objects (e.g., manufacturing tools such as robots) may be practically difficult for people who do not have the appropriate skills to do so, since spatially reconfigurable objects may have such a large number and / or very complex positions that their recognition in a real scene and their annotation manually requires a certain amount of skill and knowledge. On the other hand, a large dataset of a virtual scene can be easily obtained, for example, by easily generating the virtual scene using simulation software. Moreover, annotation of objects in a virtual scene, such as spatially reconfigurable objects, is relatively easy and, above all, can be performed automatically and without errors.Indeed, the simulation software typically knows the specifications of the objects involved in the simulation, and therefore their annotation can be performed automatically by the software, reducing the risk factor of annotation errors.

[0033] In examples, each virtual scene in the dataset of virtual scenes is a virtual manufacturing scene with one or more spatially reconfigurable manufacturing tools. In these examples, the domain adaptive neural network is configured for inference of spatially reconfigurable manufacturing tools in real manufacturing scenes. Additionally or alternatively, each real scene in the test dataset is a real manufacturing scene with one or more spatially reconfigurable manufacturing tools. In such cases, the domain adaptive neural network is configured for inference of spatially reconfigurable manufacturing tools in real manufacturing scenes.

[0034] In such examples, a domain-adaptive neural network is in each case configured for the inference of a spatially reconfigurable manufacturing tool (e.g., a spatially reconfigurable manufacturing robot) in a real manufacturing scene. The aforementioned improvements brought about by the domain adaptation learning method are particularly highlighted in these examples. In particular, it is difficult to rely on a large training set of real manufacturing scenes including spatially reconfigurable manufacturing tools, since such sets are typically confidential and / or public, but poorly annotated. On the other hand, providing a large dataset of a virtual manufacturing scene including a spatially reconfigurable manufacturing tool as a training set is particularly convenient, since such a dataset can be easily obtained and automatically annotated, for example, by using software capable of generating a virtual simulation in a virtual manufacturing environment.

[0035] The present invention also provides a domain adaptive neural network that can be trained according to the domain adaptation training method.

[0036] The present invention also provides a second computer-implemented method of machine learning. The second method includes providing a test dataset of a scene. The test dataset belongs to a test domain. The second method includes providing a domain adaptation neural network. The domain adaptation neural network is a machine learning neural network trained on data obtained from a training domain. The domain adaptation neural network is configured for inference of spatially reconfigurable objects in a scene of the test domain. The second method further includes determining an intermediate domain. The intermediate domain is closer to the training domain in terms of data distribution than the test domain. The second method further includes inferring spatially reconfigurable objects from the scene of the test domain transferred on the intermediate domain by applying the domain adaptation neural network. This second method may be referred to as a "domain adaptation inference method".

[0037] Domain adaptation inference methods form an improved method of machine learning using datasets of scenes containing spatially reconfigurable objects.

[0038] In particular, the domain adaptive reasoning method allows the inference of spatially configurable objects from a scene, which allows the domain adaptive reasoning method to be used in the process of inference of spatially reconfigurable objects, such as the process of inference of spatially reconfigurable manufacturing tools (e.g., industrial articulated robots) in a manufacturing scene.

[0039] Therefore, the improvement brought by the domain adaptation inference method is particularly highlighted in the cases where the test domain and the training domain are relatively far from each other. This is the case, for example, when the training domain is made of a virtual scene and the test domain is made of a real scene. This is also the case, for example, when both the training domain and the test domain are made of a virtual scene, but the scene of the test domain is more photorealistic than the scene of the training domain. In all these cases, the method includes (for example, in the testing phase of the domain adaptation neural network) a stage of pre-processing of the scene of the test domain provided as input to the domain adaptation neural network, i.e., the determination of an intermediate domain, which makes it closer to the scene of the training domain in terms of photorealism (which may be quantified in terms of data distribution). As a result, the level of photorealism of the scene provided as input to the domain adaptation neural network approaches the photorealism of the scene on which the domain adaptation neural network was trained. This ensures a better quality of the output of the domain adaptation neural network (i.e., the spatially reconstructable object inferred on the scene of the test domain).

[0040] Therefore, the improvement brought by the domain adaptation inference method is particularly highlighted in the cases where the test domain and the training domain are relatively far from each other. This is the case, for example, when the training domain is made of a virtual scene and the test domain is made of a real scene. This is also the case, for example, when both the training domain and the test domain are made of a virtual scene, but the scene of the test domain is more photorealistic than the scene of the training domain. In all these cases, the method includes (for example, in the testing phase of the domain adaptation neural network) a stage of pre-processing of the scene of the test domain provided as input to the domain adaptation neural network, i.e., the determination of an intermediate domain, which makes it closer to the scene of the training domain in terms of photorealism (which may be quantified in terms of data distribution). As a result, the level of photorealism of the scene provided as input to the domain adaptation neural network approaches the photorealism of the scene on which the domain adaptation neural network was trained. This ensures a better quality of the output of the domain adaptation neural network (i.e., the inferred spatially reconstructable object inferred on the scene of the test domain).

[0041] In an embodiment, the training domains include a training dataset of virtual scenes each including one or more spatially reconfigurable objects. Further, each virtual scene in the dataset of virtual scenes may be a virtual manufacturing scene including one or more spatially reconfigurable manufacturing tools, in which case the domain-adaptive neural network is configured for inference of the spatially reconfigurable manufacturing tools in the real manufacturing scene. Additionally or alternatively, the test dataset includes a real scene. In an example of such a case, each real scene in the test dataset is a real manufacturing scene including one or more spatially reconfigurable manufacturing tools, in which the domain-adaptive neural network is configured for inference of the spatially reconfigurable manufacturing tools in the real manufacturing scene.

[0042] In these examples, the domain-adaptive neural network may in any case be configured for inference of a spatially reconfigurable manufacturing tool (e.g., a spatially reconfigurable manufacturing robot) in a real manufacturing scene. The aforementioned improvements brought about by the domain-adaptive inference method are particularly highlighted in these examples. In particular, it is difficult to rely on a large training set of a real manufacturing scene including a spatially reconfigurable manufacturing tool, since such sets are typically confidential and / or public, but poorly annotated. On the other hand, using a large dataset of a virtual manufacturing scene including a spatially reconfigurable manufacturing tool as a training set is particularly convenient, since such datasets can be easily obtained and automatically annotated, for example, by using software capable of generating virtual simulations in a virtual manufacturing environment. For at least these reasons, the domain-adaptive neural network may be pre-trained on a training set of a virtual manufacturing scene. Therefore, the aforementioned pre-processing of the test dataset performed by the determination of the intermediate domain takes into account the training set of the domain adaptation neural network by judiciously bringing the real manufacturing scenes of the test dataset closer to the scenes of the training set in terms of data distribution, in order to ensure better output quality of the domain adaptation neural network.

[0043] The domain adaptation inference method and the domain adaptation training method can be performed independently. In particular, the domain adaptation neural network provided according to the domain adaptation inference method may have been trained according to the domain adaptation training method or by any other machine learning method. Providing the domain adaptation neural network according to the domain adaptation inference method may include remotely accessing, e.g., via a network, a data storage medium in which the domain adaptation neural network is stored after its training (e.g., according to the domain adaptation training method) and retrieving the domain adaptation neural network from a database.

[0044] In examples, the data acquired from the training domain includes a scene of another intermediate domain. In these examples, the domain adaptation neural network has been trained in another intermediate domain. The another intermediate domain is closer to the intermediate domain than the training domain in terms of data distribution. In such examples, the domain adaptation neural network may be a domain adaptation neural network trainable according to a domain adaptation learning method. The test data set according to the domain adaptation learning method may be equal to the test data set according to the domain adaptation inference method, and the test domain according to the domain adaptation inference method may be equal to a second domain according to the domain adaptation learning method. The training domain according to the domain adaptation inference method may be equal to a first domain according to the domain adaptation learning method, and the data acquired from the training domain may belong to (e.g., form) a third domain determined according to the domain adaptation learning method. The another intermediate domain may be the third domain determined according to the domain adaptation learning method.

[0045] Such an example combines the aforementioned improvements provided by the domain adaptation learning method and the domain adaptation inference method, namely the use of a large dataset of correctly annotated virtual scenes as a training set for the domain adaptation neural network, a pre-processing step of the training set now performed by the determination of a third domain (or another intermediate domain) that improves the quality and / or accuracy of the output of the domain adaptation neural network, and a pre-processing step of a test dataset of real scenes now performed by the determination of an intermediate domain that further improves the quality and / or accuracy of the output of the domain adaptation neural network.

[0046] The domain adaptive learning method and the domain adaptive reasoning method may alternatively be integrated into the same computer-implemented process. Figure 1 shows a flow chart illustrating the process described herein.

[0047] The process includes an offline phase of integrating a domain adaptation learning method. The offline phase includes a step S10 of providing a dataset of virtual scenes and a test dataset of real scenes according to the domain adaptation learning method. The dataset of virtual scenes belongs to a first domain, which forms a training domain of a domain adaptation neural network trained according to the domain adaptation learning method (theoretically as described above). The test dataset of real scenes belongs to a second domain, which forms a test domain of a domain adaptation neural network trained according to the domain adaptation learning method. The offline phase further includes a step S20 of determining a third domain according to the domain adaptation learning method. The third domain is closer to the second domain than the first domain in terms of data distribution. The offline phase further includes a step S30 of training the domain adaptation neural network according to the domain adaptation learning method.

[0048] Thus, after the offline phase, the process provides a domain-adapted neural network that has been trained according to the domain adaptation learning method. The domain-adapted neural network may in turn form the output of the offline phase. The offline phase may be followed by a step of storing the trained domain-adapted neural network, for example in a data storage medium of the device.

[0049] The process further includes an online phase integrating a domain adaptation inference method. The online phase includes a step S40 of providing a domain adaptation neural network trained during the offline phase according to the domain adaptation inference method. In particular, the step S40 of providing a domain adaptation neural network may include remotely accessing, for example via a network, a data storage medium in which the domain adaptation neural network is stored at the end of the offline phase and retrieving the domain adaptation neural network from a database. The online phase further includes a step S50 of determining an intermediate domain according to the domain adaptation inference method. The intermediate domain is closer to the first domain than the second domain in terms of data distribution. It should be understood that in the context of the process, the first domain of the domain adaptation learning method is a training domain of the domain adaptation inference method, the second domain of the domain adaptation learning method is a test domain of the domain adaptation inference method, the test data set of the domain adaptation learning method is a test data set of the domain adaptation inference method, and the data obtained from the training domain adaptation inference method belong to (for example form) a third domain. The online phase may further include a step S60 of estimating a spatially reconfigurable object according to the domain adaptation inference method.

[0050] This process combines the aforementioned improvements brought about by the domain adaptation learning method and the domain adaptation inference method, i.e. the use of a large dataset of correctly annotated virtual scenes as a training set for the domain adaptation neural network, a pre-processing step of the training set currently performed by the determination of a third domain, which improves the quality and / or accuracy of the output of the domain adaptation neural network, and a pre-processing step of a test dataset of real scenes currently performed by the determination of an intermediate domain, which further improves the quality and / or accuracy of the output of the domain adaptation neural network.

[0051] The process may constitute an inference process of a spatially reconfigurable manufacturing tool (e.g., a spatially reconfigurable manufacturing robot) in a real manufacturing scene. In practice, in the examples of the process, the virtual scene of the first domain can be each virtual manufacturing scene including one or more spatially reconfigurable manufacturing tools, and the real scene of the second domain can be each real manufacturing scene including one or more spatially reconfigurable manufacturing tools. Thus, the domain-adaptive neural network learned during the offline phase is configured in these examples for the inference of a spatially reconfigurable manufacturing tool in a real manufacturing scene. The offline phase benefits from the aforementioned improvements provided by the domain adaptation learning method in the specific case of the inference of a manufacturing tool, since the virtual manufacturing scenes are easy to obtain in large quantities and are correctly annotated, while the aforementioned pre-processing of the training set allows the scenes of the training set to be closer to the real scenes of the test data set, which improves the robustness and accuracy of the learned domain-adaptive neural network. Combined with these improvements are those brought about by the domain adaptation inference method integrated in the online phase of the process. These improvements include the aforementioned pre-processing of the test data set so as to bring the real scenes of the test data set closer to the scenes of the third domain on which the domain adaptation neural network was trained in the offline phase, thereby improving the quality of the output of the domain adaptation neural network. As a result, the step S60 of estimating the spatially reconfigurable manufacturing tool in the real manufacturing scene in the online phase is particularly accurate and robust. This process can therefore constitute a particularly robust and accurate process of inference of the spatially reconfigurable manufacturing tool in the real manufacturing scene.

[0052] The domain adaptive learning method, domain adaptive reasoning method and process are computer implemented. The following describes some concepts of the computer implemented method (or process).

[0053] "A method (or process) is computer-implemented" means that the steps (or substantially all steps) of the method (or process) are performed by at least one computer, or any similar system. Thus, the method steps (or process) are performed by a computer, possibly fully automatically, or semi-automatically. In an example, triggering of at least some of the steps of the method (or process) may be performed via user / computer interaction. The level of user / computer interaction required depends on the level of automation expected and may be balanced against the need to implement user wishes. In an example, this level may be user-defined and / or predefined.

[0054] A typical example of a computer implementation of the method (or process) is to execute the method (or process) using a system adapted for this purpose. The system includes a processor coupled to a memory and a graphical user interface (GUI), the memory being recorded with a computer program including instructions for carrying out the method. The memory may also store a database. The memory is any hardware adapted for such storage, and may optionally comprise several physically distinct parts (e.g. a part for the program and possibly a part for the database).

[0055] FIG. 5 shows an example of a GUI for a system, the system being a CAD system.

[0056] The GUI 2100 may be a typical CAD-like interface with standard menu bars 2110, 2120, and bottom and side toolbars 2140, 2150. Such menu bars and toolbars include a set of user-selectable icons, each icon associated with one or more operations or functions, as known in the art. Some of these icons are associated with software tools adapted to edit and / or work with the 3D modeled object 2000 displayed in the GUI 2100. The software tools may be grouped into workbenches. Each workbench includes a subset of software tools. In particular, one of the workbenches is an editing workbench suitable for editing geometric features of the modeled product 2000. During operation, the designer may, for example, pre-select a part of the object 2000 and then edit operations (e.g., change dimensions, color, etc.) or geometric constraints by selecting the appropriate icon. For example, a typical CAD operation is modeling a stamping or folding of the 3D modeled object displayed on the screen. The GUI may, for example, display data 2500 related to the displayed product 2000. In the illustrated example, the data 2500, displayed as a "feature tree", and their 3D representation 2000, relate to a brake assembly including a brake caliper and disc. The GUI may further show various types of graphic tools 2130, 2070, 2080 for facilitating, for example, 3D orientation of objects, for triggering a simulation of the operation of the edited product, or for rendering various attributes of the displayed product 2000. A cursor 2060 may be controlled by a haptic device to allow the user to interact with the graphic tools.

[0057] FIG. 6 shows an example of a system where the system is a client computer system, eg, a user's workstation.

[0058] The client computer in this example comprises a central processing unit (CPU) 1010 connected to an internal communications bus 1000, and a random access memory (RAM) 1070 also connected to the bus. The client computer is further provided with a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the BUS. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM disks 1040. Any of the foregoing may be supplemented by or incorporated in specially designed ASICs (application specific integrated circuits). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device, a haptic device 1090 such as a keyboard. A cursor control device is used in the client computer to allow the user to selectively position a cursor at any desired location on the display 1080. In addition, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a number of signal generators for inputting control signals to the system. Typically, the cursor control device may be a mouse, with the buttons of the mouse being used to generate the signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0059] A computer program may include instructions executable by a computer, the instructions including means for causing the system described above to perform the domain adaptive learning method, the domain adaptive reasoning method, and / or the process. The program may be recordable in any data storage medium, including the memory of the system. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. The program may be implemented as an apparatus, for example, an article of manufacture tangibly embodied in a machine-readable storage device for execution by a programmable processor. The process / method steps (i.e., the domain adaptive learning method, the domain adaptive reasoning method, and / or the process steps) may be performed by a programmable processor executing a program of instructions to perform the functions of the process by operating on input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high level procedural or object oriented programming language, or in assembly language or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program, and application of the program to the system will in any case result in instructions for performing the domain adaptive learning methods, domain adaptive reasoning methods, and / or processes.

[0060] The concept of providing a dataset of scenes involved in the domain adaptive learning methods, domain adaptive reasoning methods, and processes is described. Prior to the discussion of the concept, the data structures involved therein are now discussed. As will be appreciated, the data structure definitions and examples provided herein can be applied to at least a portion (e.g., all) of any dataset that provides the domain adaptive learning methods, domain adaptive reasoning methods, and / or processes.

[0061] In the context of this disclosure, a scene specifies the arrangement (e.g., placement) of one or more objects in a background at a point in time. The background typically represents a real-world physical environment, the objects typically represent real-world physical objects, and the arrangement of objects typically represents the arrangement of real-world objects in the real-world environment at a particular point in time. This is equivalent to saying that a scene is a representation of real-world objects in their real-world environment at a point in time. This representation may be real, geometrically realistic, semantically realistic, photorealistic, and / or virtual. The concepts of real, geometrically realistic, semantically realistic, photorealistic, and virtual scenes are explained below. A scene is typically (but not always) three-dimensional. In the following, no distinction is made between a scene (e.g., virtual or real) and its representation (e.g., by virtual or real images) for the sake of simplicity.

[0062] In the context of this disclosure, a dataset of scenes typically includes a large number of scenes, e.g., more than 1000, 10000, 100000, or 1000000 scenes. Any dataset of scenes of this disclosure may be created or substantially created from scenes from the same context (e.g., manufacturing scenes, construction site scenes, harbor scenes, or tarmac scenes). The scenes included in the domain adaptive learning method, domain adaptive reasoning method, and / or process may in fact all be scenes forming the same context (e.g., manufacturing scenes, construction site scenes, harbor scenes, or tarmac scenes).

[0063] Any scene included in the domain adaptive learning method, domain adaptive reasoning method and / or process may include one or more spatially reconfigurable objects. A spatially reconfigurable object is an object that includes several (e.g., mechanical) parts, the object being characterized in that there exists one or more semantically realistic spatial relationships between said parts of the object. A spatially reconfigurable object may typically be an (e.g., mechanical) assembly of parts physically connected to each other, the assembly having at least one degree of freedom. This means that at least a first part of the object can move relative to at least a second part of the object, while both the at least first part and the at least second part remain physically attached to the assembly. For example, the at least a first part may be translated and / or rotated relative to the at least a second part. Thus, a spatially reconfigurable object may have a very large number of spatial configurations and / or positions, which makes it difficult for a person without appropriate skills to identify, e.g., label, the object. In the context of this disclosure, a spatially reconfigurable object may be a spatially reconfigurable manufacturing tool (such as an industrial articulated robot) in a manufacturing scene, a crane in a construction site scene, an air gangway in a tarmac scene, or a port crane in a harbor scene.

[0064] In the context of the present disclosure, a scene may be a manufacturing scene. A manufacturing scene is a scene that represents real-world objects in a real-world manufacturing environment. The manufacturing environment may be a factory or part of a factory. The objects in the factory or part of a factory may be manufactured products, i.e. products that have already been manufactured by one or more manufacturing processes performed in the factory at a time relevant to the scene. The objects in the factory or part of a factory may also be manufactured products, i.e. products that are to be manufactured by one or more manufacturing processes performed in the factory at a time relevant to the scene. The objects in the factory or part of a factory may also be manufacturing tools (e.g., spatially reconfigurable), that is, tools involved in one or more manufacturing processes performed in the factory. The objects in the factory may also be objects that make up the background of the factory.

[0065] In other words, a scene representing a factory or part of a factory may thus include one or more products, each being or already being produced by one or more manufacturing processes of the factory, and, for example, one or more manufacturing tools, each involved in the production of one or more of said products. A product may typically be a (e.g., mechanical) part or assembly of parts (or equivalently an assembly of parts, since an assembly of parts may itself be viewed as a part in terms of this disclosure).

[0066] The mechanical parts may be parts of land vehicles (including, for example, automobiles and light truck equipment, racing cars, motorcycles, trucks and motor equipment, trucks and buses, trains), parts of aircraft (including, for example, airframe equipment, aerospace equipment, propulsion equipment, defense equipment, aircraft equipment, space equipment), parts of naval vehicles (including, for example, naval equipment, commercial vessels, offshore equipment, yachts and workboats), general mechanical parts (including, for example, industrial manufacturing machinery, heavy mobile machinery or equipment, installation equipment, industrial equipment, fabricated metal products, tire manufacturing products, articulated and / or reconfigurable manufacturing equipment such as robotic arms), electric machinery or electronic parts (including, for example, consumer electronics, security and / or control and / or instrumentation products, computer and communication equipment, semiconductors, medical devices and equipment), consumer goods (including, for example, furniture, home and garden products, leisure products, fashion products, hard goods retailer products, soft goods retailer products, packaging (including, for example, food and beverage and tobacco, beauty and personal care, household goods packaging)).

[0067] The mechanical part may also be one or any suitable combination of a molded part (i.e., a part produced by a molding manufacturing process), a machined part (i.e., a part produced by a machining manufacturing process), a punched part (i.e., a part produced by a punching manufacturing process), a turned part (i.e., a part produced by a turning manufacturing process), a forged part (i.e., a part produced by a forging manufacturing process), a stamped part (i.e., a part produced by a stamping manufacturing process), and / or a folded part (i.e., a part produced by a folding manufacturing process).

[0068] The manufacturing tool may be any of the following: Machining tools (i.e., tools that perform at least a part of a machining process) such as broaches, drill presses, gear shapers, hobbing machines, hones, lathes, turning machines, milling machines, sheet metal shears, shapers, saws, planers, Stewart platform mills, grinding machines, multi-task machines (e.g., having many axes that combine turning, milling, grinding, and / or material handling into one highly automated machine tool). A compression molding machine (i.e., a machine that performs at least a portion of a compression molding process), typically including at least one die or molding matrix, such as a flush plunger type, a straight plunger type, or a land plunger type die. Injection molding machines (i.e. machines that perform at least part of an injection molding process) such as die casting machines, metal injection molding machines, plastic injection molding machines, liquid silicone injection molding machines or reaction injection molding machines. A rotary cutting tool, such as a drill bit, countersink, counterbore, tap, die, milling cutter, reamer, or cold saw blade (e.g., to perform at least a portion of the drilling operation). A non-rotating cutting tool (eg, performing at least part of the turning process), such as a tipped tool or a forming tool. Forging machines (i.e., perform at least part of the forging process), such as mechanical forging presses (typically including at least one forging die), hydraulic forging presses (typically including at least one forging die), compressed air powered forging hammers, electrically powered forging hammers, hydraulically powered forging hammers, or steam powered forging hammers. A stamping machine (i.e. performing at least part of the stamping process) such as a stamping press, press, punching machine, blanking machine, embossing machine, bending machine, flanging machine or coining machine. Bending or folding machines (i.e., which perform at least part of the bending or folding process), such box and pan brakes, press brakes, folders, panel benders or mechanical presses. · Electro-painting tools (i.e. performing at least part of the painting process of a product) such as spot welding robots or (e.g. electric) automotive painting robots.

[0069] Any one of the manufacturing tools listed above may be a spatially reconfigurable manufacturing tool, i.e., a spatially reconfigurable manufacturing tool, or may form at least a portion of a spatially reconfigurable manufacturing tool. For example, a manufacturing tool held by a tool holder of a spatially reconfigurable manufacturing tool forms at least a portion of a spatially reconfigurable manufacturing tool. A spatially reconfigurable manufacturing tool (e.g., any one of the tools listed above) may be a manufacturing robot, such as an (e.g., articulated) industrial robot.

[0070] The concept of a real scene is now described. As mentioned above, a scene represents an arrangement of real-world objects in a real-world environment (i.e., the background of the scene). A scene is real when the representation of the arrangement is a physical arrangement resulting from the acquisition of the same physical arrangement as in the real world. The acquisition may be performed by a digital image acquisition process, which generates a digitally encoded representation of the visual features of the physical arrangement of objects in the environment. Such a digitally encoded representation may be referred to as an "image" or a "digital image" hereinafter for brevity. The digital image acquisition process may optionally include processing, compression, and / or storage of such an image. A digital image may be created directly from the physical environment by a camera or similar device, a scanning or similar device, and / or a video camera or similar device. Alternatively, a digital image may be acquired by an image scanner or similar device from another image in a photograph, a photographic film (e.g., a digital image is a snapshot of an instant on a film), or a similar medium such as printed paper. Digital images may also be obtained by processing non-image data, such as those obtained using tomography, scans, side-scan sonar, radio telescopes, X-ray detectors and / or photostimulable phosphor screens (PSPs). In the context of the present disclosure, the real manufacturing scene may originate, for example, from digital images acquired by cameras (or similar devices) located in the factory. Alternatively, the manufacturing scene may originate from film snapshots captured by one or more video cameras located in the factory.

[0071] 7 to 12 show examples of actual production scenes, each of which is made up of articulated industrial robots 70, 72, 80, 90, 92, 94, 96, 98, 100, 102, 104, 110, 112, 120, 122, and 124.

[0072] We now turn to the concept of a virtual scene. A scene is virtual if it originates from computer-generated data. Such data may be generated by (e.g., 3D) simulation software, (e.g., 3D) modeling software, CAD software, or video games. This means that the scene itself is computer-generated. The objects of the virtual scene may be placed in realistic positions in the virtual scene. Alternatively, the objects of the virtual scene may be placed in unrealistic positions in the virtual scene, for example, as described below, among others, when the objects are randomly placed in an existing virtual scene. This means that the virtual scene represents an arrangement of virtual objects (e.g., virtual representations of real-world objects) in a virtual background (e.g., a virtual representation of a real-world environment). The arrangement may be physically realistic for at least some of the virtual objects, i.e. the arrangement of the at least some virtual objects represents a real-world physical arrangement of real-world objects. Additionally or alternatively, at least another portion of the virtual objects may be arranged (e.g., relative to each other and / or to the at least some objects) in an unrealistic arrangement (i.e., not corresponding to a real-world physical arrangement), e.g., the at least some virtual objects are randomly arranged or positioned, as described in particular below. Any object in the virtual scene may be automatically annotated or labeled. The concept of labeling objects is further described below. The virtual manufacturing scene may be generated by 3D modeling software for a manufacturing environment, 3D simulation software for a manufacturing environment, or CAD software. Any object in the virtual manufacturing scene (such as a product or a manufacturing tool) may be virtually represented by a 3D modeled object representing the skin (e.g., outer surface) of a solid (e.g., a B-rep model). The 3D modeled object may be designed by a user with a CAD system and / or may originate from a virtual manufacturing simulation performed by 3D simulation software for a manufacturing environment.

[0073] A modeled object is any object defined by data stored, for example in a database. By extension, the expression "modeled object" designates the data itself. Depending on the type of system in which the domain adaptive learning method, the domain adaptive reasoning method, and / or the processing are performed, the modeled object may be defined by different kinds of data. The system may in fact be any combination of a CAD system, a CAE system, a CAM system, a PDM system, and / or a PLM system. In these different systems, the modeled object is defined by the corresponding data. Thus, one can speak of CAD objects, PLM objects, PDM objects, CAE objects, CAM objects, CAD data, PLM data, PDM data, CAM data, CAE data. However, these systems do not make one of the systems exclusive of the other, since the modeled object may be defined by data corresponding to any combination of these systems. Thus, a system may be both a CAD system and a PLM system, as will become clear from the definition of such systems provided below.

[0074] A CAD system further means any system adapted to at least design a modeled object based on a graphical representation of the modeled object, such as CATIA. In this case, the data defining the modeled object includes data enabling the representation of the modeled object. A CAD system may, for example, provide a representation of a CAD modeled object using edges or lines, in some cases with faces or surfaces. The lines, edges or surfaces may be represented in various ways, for example with Non-Uniform Rational B-Splines (NURBS). In particular, a CAD file contains specifications from which a geometric shape may be generated, and thus a representation may be generated. The specifications of the modeled object may be stored in a single CAD file or in multiple CAD files. A typical size of a file representing a modeled object in a CAD system is in the range of 1 megabyte per part. And the modeled object may typically be an assembly of thousands of parts.

[0075] CAM solution further means any solution, hardware or software, adapted to manage the manufacturing data of a product. The manufacturing data generally includes data about the product to be manufactured, the manufacturing process and the necessary resources. CAM solutions are used to plan and optimize the entire manufacturing process of a product. For example, the CAM user can be provided with information about the feasibility of the manufacturing process, its duration or the number of resources, such as specific robots, that can be used at a particular step of the manufacturing process, thus allowing decisions regarding management or necessary investments. CAM is a follow-up process after the CAD process and potentially the CAE process. Such CAM solutions are offered by Dassault Systèmes under the trademark DELMIA®.

[0076] In the context of CAD, a modeled object may typically be, for example, a 3D modeled object that represents a product, such as a part or an assembly of parts, or possibly an assembly of a product. By "3D modeled object" is meant any object that is modeled by data that allows for its 3D representation. The 3D representation allows for the part to be viewed from all angles. For example, a 3D modeled object, when represented in 3D, may be manipulated and rotated around any of its axes, or around any axis within the screen on which the representation is displayed. This notably excludes 2D icons that are not 3D modeled. The display of 3D representations facilitates design (i.e. increases the speed at which designers statistically accomplish their tasks). This speeds up the manufacturing process in industry, since the design of the product is part of the manufacturing process.

[0077] 13 to 17 show virtual manufacturing scenes comprising virtual articulated industrial robots 130, 140, 150, 152, 160, 162, 170, and 172, respectively.

[0078] A virtual scene may be geometrically and semantically realistic, meaning that the geometry and functionality of virtual objects in a scene (such as products or manufacturing tools in a manufacturing scene) are realistic, e.g., realistically represent the corresponding geometry and functionality as in the real world. A virtual scene may be geometrically and semantically realistic without being photorealistic. Alternatively, a virtual scene may be both geometrically and semantically realistic and photorealistic.

[0079] The concept of a photorealistic image can be understood through a number of visible features, such as shading (i.e., how the color and brightness of a surface changes with lighting), texture mapping (i.e., how to apply detail to a surface), bump mapping (i.e., how to simulate small scale unevenness on a surface), fogging / participating media (i.e., how light darkens when passing through an opaque atmosphere or air), shadows (i.e., the varying darkness caused by a partially hidden light source), reflection (i.e., a specular, very glossy reflection), transparency or opacity (i.e., the sharp transmission of light through a three-dimensional object), refraction (i.e., the bending of light associated with transparency), and diffraction (i.e., the effect of light reflecting off of a solid object). bending, interference of light passing through objects or apertures that diffuse and disrupt light rays, indirect illumination (i.e., interference of light that is not directly illuminated by a light source but by light reflected from other surfaces), ablation (i.e., a form of indirect illumination, reflection of light from shiny objects, and / or focusing of light through transparent objects), producing bright highlights on another object, depth of field (i.e., objects appear blurry or out of focus when they are too far in front of or behind an object in focus), and / or motion blur (i.e., objects appear blurry due to rapid motion or camera movement). The presence of such properties causes such data (i.e., corresponding to or resulting from a virtual photorealistic scene) to have a distribution similar to real data (arriving from an image representing a real scene). In this context, the closer the virtual ensemble distribution is to the real-world data, the more realistic it appears.

[0080] We now introduce the concept of data distribution. In the context of building machine learning (ML) models, it is often desirable to train the model and test it against data that originates from the same distribution. In probability theory and statistics, a probability distribution is a mathematical function that provides the probability of occurrence of various possible outcomes in an experiment. In other words, a probability distribution is a description of a random phenomenon in terms of the probability of an event. A probability distribution is defined with respect to an underlying sample space with the set of all possible outcomes of the random phenomenon being observed. The sample space may be a set of real numbers, a higher dimensional vector space, or a non-numeric list. It can be seen that during training on virtual data and testing on real data, the distribution shift problem is faced. When correcting the virtual image with the real image to address the shift problem, domain adaptive learning methods, domain adaptive inference methods, and / or processes can transform the distribution of the virtual data to better align with the distribution of the real data. To quantify how well the two distributions align, several approaches can be used. For illustrative purposes, two examples of approaches are given below.

[0081] The first approach is the histogram approach. Consider creating N image pairs, each consisting of a first generated virtual image and a second virtual image corresponding to a real image depicting the same scene as the first virtual image, where the real images are obtained by using a generator virtual-real generator. To evaluate the relevance of modifying the training data, the domain adaptation learning method, domain adaptation inference method, and / or process quantify how well the virtual image distribution aligns with the real-world data before and after the synthetic-to-real modification. To do so, the domain adaptation learning method, domain adaptation inference method, and / or process may consider a set of real images and calculate the Euclidean distance between the average image of the real data and the virtual data before and after the synthetic-to-real transformation. Formally, deep features derived from a pre-trained network, e.g., trained on the ImageNet classification task (Inception network, see

[22] ) of each virtual image before and after the transformation, may be used to estimate the relevance of the training data.

number

number

number

number

number

number

number

[0082] The second approach is the Fréchet Inception Distance (FID) approach (see

[21] ). FID compares the distributions of Inception embeddings (the activations from the final layer of Inception, see

[22] ) of two sets of images. These distributions are modeled as multidimensional Gaussian distributions parameterized by their respective means and covariances. Formally,

number

number

number

number

[22] ). Then, the second approach is

number

number

number

number

number

[0083] The smaller the distance,

number

[0084] From the previous discussion on data distribution, it follows that the distance between two datasets of a scene can be quantified by a distance, such as the Euclidean distance or FID discussed previously. Thus, in the context of the present disclosure, "a first dataset (e.g., family, e.g., domain) of a scene has a concept of the domain of the scene closer to a second dataset (e.g., family, e.g., domain) of the scene than a third dataset (e.g., family, e.g., domain) of the scene" means that the distance between the first dataset (family or domain, respectively) and the second dataset (family or domain, respectively) is smaller than the distance between the third dataset (family or domain, respectively) and the second dataset (family or domain, respectively). If the second dataset is a dataset of a real scene and both the third and the first dataset are datasets of virtual scenes, then the first dataset is said to be more photorealistic than the third dataset.

[0085] Here, the concept of "domain" is explained. A domain specifies the distribution of a family of scenes. Thus, a domain is a family of distributions as well. A dataset of scenes belongs to a domain if the scenes of the dataset and the scenes of the family are close in terms of data distribution. In an example, there is a predefined threshold such that the distance (e.g., Euclidean or FID as described above) between the scenes of the dataset and the scenes of the family is smaller than the predefined threshold. It should be understood that the dataset of scenes may form such a domain, in which case the domain may be referred to as the "domain of scenes" or the "domain made from the scenes of the dataset". The first domain according to the domain adaptation learning method is a domain of virtual scenes, for example a domain of virtual scenes, each of which is geometrically and semantically realistic, but not photorealistic. The second domain according to the domain adaptation learning method is a domain of real scenes. The test domain according to the domain adaptation learning method may be a domain of real scenes. Alternatively, the test domain may be a domain of virtual scenes, for example, which are geometrically and semantically realistic, and optionally photorealistic. The training domain of the domain adaptation learning method may for example be a domain of geometrically and semantically realistic, and optionally photorealistic, virtual scenes, in an example where the training domain is a domain of virtual scenes that are less photorealistic than the scenes in the test domain, e.g. the training domain is a domain of virtual scenes and the test domain is a domain of real scenes.

[0086] Next, the concept of providing a scene dataset is explained.

[0087] In the context of the present disclosure, providing a dataset of a scene may be performed automatically or by a user. For example, the user may retrieve the dataset from a memory where the dataset is stored. Furthermore, the user may choose to complete the dataset by adding one or more scenes (e.g., one by one) to an already acquired dataset. Adding one or more scenes may include retrieving them from one or more memories and including them in the dataset. Alternatively, the user may create at least a part (e.g., all) of the scenes of the dataset. For example, when providing a dataset of a virtual scene, the user may first retrieve at least a part (e.g., all) of the scene by using 3D simulation software and / or 3D modeling software and / or CAD software, as discussed above. More generally, providing a dataset of a virtual scene may be preceded in the context of a domain adaptive learning method, a domain adaptive reasoning method and / or a process by a step of retrieving (e.g., calculating) the scene, for example, by using 3D simulation software (e.g., for a manufacturing environment) and / or 3D modeling software and / or CAD software, as discussed above. Similarly, in the context of domain adaptive learning methods, domain adaptive inference methods and / or processes, providing a dataset of a real scene may be preceded by a step of acquiring (e.g., acquiring) the scene, e.g., via a digital image acquisition process, as described above.

[0088] In examples of the domain adaptation learning method and / or process, each virtual scene of the dataset of virtual scenes is a virtual manufacturing scene including one or more spatially reconfigurable manufacturing tools. In these examples, the domain-adaptive neural network is configured for inference of spatially reconfigurable manufacturing tools in real manufacturing scenes. In these examples, each virtual scene of the dataset of virtual scenes may result from a simulation. Thus, the step S10 of providing a dataset of virtual scenes may precede a step of performing a simulation, for example using 3D simulation software for manufacturing environments, as described above. In examples, the simulation may be a 3D experience (i.e. a virtual scene with accurate modeling and with faithful (e.g., matching reality) behavior), where different configurations of a given scene are simulated, for example different arrangements of objects in a given scene, and / or different configurations of several (e.g., spatially reconfigurable) objects in a given scene. The dataset may in particular include virtual scenes corresponding to said different configurations.

[0089] In examples of domain adaptation learning methods and / or processes, each real scene of the test data set is a real manufacturing scene including one or more spatially reconfigurable manufacturing tools. In these examples, the domain-adaptive neural network is configured for inference of the spatially reconfigurable manufacturing tools in the real manufacturing scenes. In these examples, each real scene of the test data set may originate from an image acquired in a digital acquisition process. Thus, before the step S10 of providing the test data set of the virtual scene, a step of acquiring images of the factory from which the scenes of the test data set originate, for example by using one or more scans and / or one or more cameras and / or one or more video cameras, may be performed.

[0090] In examples of domain adaptation inference methods and / or processes, the test dataset comprises real scenes. In these examples, each real scene of the test dataset may be a real manufacturing scene including one or more spatially reconfigurable manufacturing tools, and the domain adaptive neural network is optionally configured for inference of the spatially reconfigurable manufacturing tools in the real manufacturing scenes. Thus, the step S10 of providing the test dataset of the virtual scene may be preceded by a step of acquiring images of the factory from which the scenes of the test dataset originate, for example by using one or more scans and / or one or more cameras and / or one or more video cameras.

[0091] Next, step S20 of determining the third domain will be described.

[0092] The step S20 of determining the third domain is performed such that, in terms of data distribution, the third domain is closer to the second domain than to the first domain. Thus, the step S20 of determining the third domain specifies, for each virtual scene of the dataset of virtual scenes belonging to the first domain, an inference of another scene such that all inferred other scenes form a domain closer to the second domain than the first domain in terms of data distribution, i.e. a third domain. The inference of said other scene may comprise a transformation (e.g. conversion) of the virtual scene into said other scene. The transformation may for example consist of transforming the virtual scene into said other scene using a generator capable of transferring images from one domain to another. It is to be understood that such a transformation concerns only one or more parts of the virtual scene, said one or more parts being transformed, for example using a generator, while the remainder of the scene remains unchanged. The said other scene may be composed of a scene of which one or more parts are transformed and the remainder of the scene is not changed. Alternatively, the transformation may comprise replacing the remainder of the scene by a part of another scene (e.g. belonging to the second domain). In such a case, the other scene may be within a scene in which one or more parts have been transformed and the remaining scene replaced by said parts of some other scene. It should be understood that inference can be performed similarly for all scenes of the first domain, e.g., using the same paradigm for all scenes.

[0093] Next, an example of step S20 of determining the third domain will be described with reference to FIG. 2, which shows a flow chart illustrating an example of step S20 of determining the third domain.

[0094] In examples, the step S20 of determining the third domain comprises, for each scene of the data set of virtual scenes, a step S200 of extracting one or more spatially reconstructable objects from the scene. In these examples, the step S20 of determining further comprises, for each extracted object, a step S210 of transforming the extracted object into an object that is closer to the second domain in terms of the data distribution than the extracted object.

[0095] As mentioned above, the virtual scenes of the virtual scene dataset may be obtained prior to step S10 of providing the virtual scene dataset by calculating a virtual simulation from which the scene originates, for example using 3D simulation software. The calculation of the simulation may include automatic localization and annotation of each spatially reconfigurable object in each virtual scene. For each virtual scene of the dataset and for each spatially reconfigurable object in the virtual scene, the localization may include the calculation of a bounding box around the spatially reconfigurable object. The bounding box is a rectangular box with axes parallel to the side of the image representing the scene. The bounding box is characterized by four coordinates and is centered on the spatially reconfigurable object at an appropriate ratio and scale so as to encompass the entire spatially reconfigurable object. In the context of the present disclosure, encompassing an object in a scene may mean encompassing all parts of the object that are fully visible on the scene, said parts being the entire object or only a part of it. The annotation may include assigning to each computed bounding box a label indicating that the bounding box encompasses the spatially reconfigurable object. It should be appreciated that the label may indicate further information about the bounding box, for example, that the bounding box encompasses a spatially reconfigurable manufacturing tool, such as an industrial robot.

[0096] For each one of the one or more spatially reconfigurable objects, the step S200 of extracting the spatially reconfigurable object from the virtual scene may specify any action that will result in isolating the spatially reconfigurable object from the virtual scene and may be performed by any method capable of performing such a separation. The step S200 of extracting the spatially reconfigurable object from the virtual scene may in fact include a step of detecting a bounding box surrounding the spatially reconfigurable object, the step of detecting being based on a label of the bounding box and optionally including a step of accessing the label. The step S200 of extracting may further include a step of isolating the bounding box from the virtual scene, thereby isolating the spatially reconfigurable object from the virtual scene.

[0097] The step S210 of transforming the extracted object can specify any transformation that changes the extracted object to another object that is closer to the second domain in terms of the data distribution. "Close to the second domain" means that the object to the extracted object is transformed into itself, which is a scene that belongs to a domain that is closer to the second domain than the first domain. If the step S200 of extracting the object includes isolating a bounding box that surrounds the object from the virtual scene as described above, the step S210 of transforming the extracted object can include a modification of the entire bounding box (and thus includes the extracted object) or only the extracted object, which only brings it closer to the second domain in terms of the data distribution. This variant can include applying to the bounding box or to the spatially reconstructable object a generator that can use the bounding box to transform the virtual image into a more photorealistic image in terms of the data distribution.

[0098] In one implementation, the transforming step S210 can be performed by a CycleGAN virtual-to-real network (see

[20] ) trained on two sets of spatially reconfigurable objects, namely a virtual set and a real set. This results in a virtual-to-real spatially reconfigurable object (e.g., robot) generator that is used to modify the spatially reconfigurable object (e.g., robot) in a virtual image. The CycleGAN model transforms images from one domain to another. The present invention proposes a method that can capture special features of one image set and understand how to transform these features into a second collection of images, in the absence of all paired training examples. This relaxation with one-to-one mapping makes this formulation very powerful in practice. This is achieved by a generative model, specifically a generative adversarial network (GAN) called CycleGAN.

[0099] 19 shows an example of the steps of extracting S200 and transforming S210 two virtual industrial articulated robots 190 and 192. Each of these robots 190 and 192 is extracted (S200) from a respective virtual scene. The extracted robots 190 and 192 are then transformed (S210) into more photorealistic robots 194 and 196, respectively.

[0100] 2, determining the third domain may further include, for each transformed extracted object, a step S220 of placing the transformed extracted object in one or more scenes that are respectively closer to the second domain than to the first domain in terms of data distribution. In such a case, the third domain includes each scene in which the transformed extracted object is placed.

[0101] Each of the one or more scenes may be a scene that does not contain spatially reconstructable objects, for example a background scene (i.e. a scene that contains a single background, although the background itself may contain background objects). Each of the one or more scenes may also be a scene from the same context as the scene to which the transformed extracted object originally belongs. This same context may in fact be the context of all scenes of the dataset of the virtual scene.

[0102] Each scene of the one or more scenes is closer to the second domain than to the first domain. "Closer to the second domain" means that each scene of the one or more scenes belongs to a domain closer to the second domain than to the first domain. In an example, each scene of the one or more scenes is a real scene taken directly from reality as described above. Alternatively, each scene of the one or more scenes may be generated from a respective virtual scene of a dataset of virtual scenes, one of said respective virtual scenes being, for example, a virtual scene to which the transformed extracted object belongs, such that the generated scene is more real and / or photorealistic than the respective virtual scene. Generating the scene from the respective virtual scene of the dataset may include applying to the virtual scene a generator capable of transforming a virtual image into a real and / or more photorealistic image. In one embodiment, the generation of the scene may be performed by a CycleGAN virtual-to-real generator (see

[20] ) trained on two sets of backgrounds (e.g., factory backgrounds), namely a virtual set and a real set. This results in a virtual-real background (e.g., factory background) generator that is used to modify backgrounds (e.g., factory backgrounds) in virtual images. The CycleGAN model (see

[20] ) translates images from one domain to another. We propose a method that can capture special features of one set of images and understand how to translate these features to a second collection of images, without the need for full paired training examples. This relaxation with a one-to-one mapping makes this formulation very powerful in practice. This is achieved by a generative model, specifically a generative adversarial network (GAN) called CycleGAN.

[0103] The step S220 of placing the transformed extracted object may be performed (e.g., automatically) by any method capable of pasting single and / or multiple images of the object onto an image of a background. Thus, the step S220 of placing said one or more scenes of the transformed extracted object may include applying such a method to paste the transformed extracted object onto a respective one of the one or more scenes. It should be understood that the scene in which the transformed extracted object is placed may contain one or more other transformed extracted objects already placed on the scene (S220).

[0104] From the above examples, it follows that in these examples the step S20 of determining the third domain specifically transforms the spatially reconfigurable objects of the scene and the background of the scene to which they belong separately, and then each of the transformed spatially reconfigurable objects can be placed in one or more of the transformed backgrounds. Firstly, this allows to obtain more configurations of the transformed objects in the backgrounds transformed completely (i.e. without separating the spatially reconfigurable objects from their backgrounds) of the virtual scene of the dataset of virtual scenes. Since the step S30 of training the domain-adapted neural network is based on the scenes of the third domain, the learning is more robust since the third domain is closer to the test domain. Secondly, transforming the entire virtual scene of the dataset of virtual scenes may lead to unsatisfactory results since there may be no underlying relationship between the scenes of the test domain due to the variability of the real background. For example, transforming the entire virtual scene including the spatially reconfigurable objects may result in a transformed scene in which the spatially reconfigurable objects cannot be distinguished from the background (e.g. may be mixed with the background). For all these reasons, completely transforming the virtual scenes of a virtual scene dataset may yield poor results, for example by generating a third domain that cannot be any closer to the second domain than the first domain. Moreover, in this case, the scenes of the third domain may not be suitable for learning, since the properties of objects of interest (spatially reconfigurable objects) in these scenes may be affected by the transformation.

[0105] Fig. 20 illustrates an example of the step S220 of placing the transformed extracted objects. Fig. 20 illustrates the transformed extracted industrial articulated robots 194 and 196 of the example of Fig. 19 placed S220 in the factory background technical scenes 200 and 202, respectively.

[0106] 2, in an example, the step S220 of placing the transformed extracted object in one or more scenes is performed randomly. Such random placing step S220 of the transformed extracted object may be performed (e.g., automatically) by any method capable of randomly pasting single and / or multiple images of the object within an image of a background. Thus, the step S220 of randomly placing the transformed extracted object may include applying such a method to randomly paste the transformed extracted object in a respective one of the one or more scenes.

[0107] In the above example, the third domain includes, for each transformed extracted object, each scene in which the transformed extracted object is placed (S220). For example, the third domain is composed of such scenes. That is, for all scenes of the dataset of virtual scenes, the steps of extracting S200, transforming S210 and placing S220 are performed in the same way. For example, they are performed such that all scenes in which one or more transformed extracted objects are placed (S220) have substantially the same level of photorealism. In other words, they belong to and / or form a domain, namely the third domain.

[0108] In the above example, the third domain includes, for each transformed extracted object, each scene in which the transformed extracted object is placed (S220). For example, the third domain is composed of such scenes. That is, for all scenes of the dataset of virtual scenes, the steps of extracting S200, transforming S210 and placing S220 are performed in the same way. For example, they are performed such that all scenes in which one or more transformed extracted objects are placed (S220) have substantially the same level of photorealism. In other words, they belong to and / or form a domain, i.e. the third domain.

[0109] 2, the step S20 of determining the third domain may include a step S230 of placing one or more distractors in a respective one of the one or more scenes included in the third domain. A distractor is an object that is not a spatially reconfigurable object.

[0110] The distractors may be objects having substantially the same level of photorealism as the transformed extracted objects. In fact, the distractors may be objects of the virtual scene of a data set of the virtual scene that are extracted from said virtual scene, transformed into objects more similar to the second domain, and placed in the scene of the third domain, similar to the spatially reconfigurable objects of the virtual scene. Alternatively, the distractors may originate from another data set of real or virtual distractors, in which case the determining step S20 may include a step of providing such a data set. In any case, it is understood that the distractors may be placed (S230) in the scene of the third domain, and that after such a placing step S230, the scene still has a level of photorealism such that it belongs to the third domain.

[0111] The distractors may be, for example, mechanical parts extracted and transformed from the virtual manufacturing scene as described above, or any other object that is, for example, part of the background of the virtual (e.g., manufacturing) scene. Figure 21 shows three distractors 210, 212, and 214 that are placed in one or more scenes of a third domain in a manufacturing scene context. The distractors 210, 212, and 214 are, respectively, a hard hat (which is an object in the background of the manufacturing scene), a metal sheet (which is a machine part), and an iron horseshoe-shaped forged part (which is a machine part).

[0112] The step S230 of placing distractors, in particular distractors obtained similarly to the transformed extracted object (as described above), can reduce the sensitivity of the neural network to numerical (e.g. digital, e.g. imaging) artifacts. Indeed, as a result of the steps S200 of extracting and / or S210 of transforming and / or S220 of placing, the scene of the third domain featuring at least one transformed extracted object can feature one or more numerical artifacts at the locations where the at least one transformed extracted object is placed (S220). The step S230 of placing distractors in the scene can cause the same kind of artifacts to appear at the locations where the distractors are placed. Thus, once trained (S30), the domain-adapted neural network does not focus on artifacts, or at least does not link them to spatially reconstructable objects, since the artifacts are also caused by the distractors. This makes the step S30 of learning more robust.

[0113] Returning to the flowchart of FIG. 1, step S30 of training the domain adaptive neural network will now be described.

[0114] We first briefly discuss the general concepts of "machine learning" and "training neural networks".

[0115] As known from the field of machine learning, the processing of an input by a neural network involves applying an operation to the input, the operation being defined by data including weight values. Thus, training a neural network involves determining the values ​​of the weights based on a data set configured for such training, such a data set may be called a learning or training data set. To that end, a data set includes pieces of data each forming a respective training sample. The training samples represent the diversity of situations in which the neural network will be used after it has been trained. Any data set referred to herein may include a number of training samples that exceeds 1000, 10000, 100000, or 1000000. In the context of the present disclosure, "training a neural network based on a data set" means that the data set is the learning / training set of the neural network, and "training a neural network based on a domain" means that the training set of the neural network is a data set that belongs to and / or forms a domain.

[0116] Any of the neural networks of this disclosure, e.g., domain adaptive neural networks, or detectors, teacher extractors, or student extractors, all of which are described below, may be deep neural networks (DNNs) trained by any deep learning technique. Deep learning techniques are a powerful set of techniques for learning in neural networks (see

[19] ), a biologically-inspired programming paradigm that allows computers to learn from observed data. In image recognition, the success of DNNs is due to their ability to learn rich mid-level media representations, as opposed to the hand-designed low-level features (Zernike moments, HOG, Bag-of-Words, SIFT) used in other image classification methods (SVM, Boosting, Random Forest). More specifically, DNNs focus on end-to-end learning based on raw data. In other words, they move away from feature engineering to the maximum extent possible by achieving end-to-end optimization starting with raw features and ending with labels.

[0117] Here, we will discuss the concept of "configured for inference of spatially reconfigurable objects in a scene". This discussion applies in particular to domain-adaptive neural networks according to any one of the domain-adaptive learning methods, domain-adaptive inference methods, or processes, independent of the manner in which the domain-adaptive neural network is trained. Inferring spatially reconfigurable objects in a scene means inferring one or more spatially reconfigurable objects in a scene that include one or more of such objects. This can mean inferring all spatially reconfigurable objects in a scene. Thus, a neural network configured to infer spatially reconfigurable objects in a scene takes the scene as input and outputs data regarding the location of one or more of the spatially reconfigurable objects. For example, the neural network can output a (e.g., digital) image representation of the scene, each bounding box respectively encompassing a respective one of the inferred spatially reconfigurable objects. Inference can also include computation of such bounding boxes, and / or labeling of the bounding boxes (e.g., as encompassing the spatially reconfigurable objects of the scene), and / or labeling of the inferred spatially reconfigurable objects (as spatially reconfigurable objects). The output of the neural network may be stored in a database and / or displayed on a computer screen.

[0118] The step S30 of training a domain-adaptive neural network may be performed by any machine learning algorithm capable of training a neural network configured for inference of spatially reconfigurable objects in real scenes.

[0119] The step S30 of training the domain-adapted neural network is based on a third domain. The third domain is therefore or at least includes the training / learning set of the domain-adapted neural network. From the previous discussion on the step S20 of determining the third domain, it is clear that the training / learning set (e.g. obtained) of the domain-adapted neural network stem is obtained from the provided S10 dataset of virtual scenes. In other words, the step S20 of determining can be considered as a pre-processing of the theoretical training / learning set, which is a dataset of virtual scenes, to transform it into a practical training / learning set, i.e. a scene of the third domain, which is closer to the set from which the neural network, once trained, gets its input. This improves the robustness of the step S30 of training and the output quality of the domain-adapted neural network, as described above.

[0120] The domain adaptive neural network may comprise (e.g. may consist of) a detector, also called an object detector, which forms at least a part of the domain adaptive neural network. The detector is thus itself a neural network and is further configured to detect spatially reconfigurable objects in a real scene. The detector may typically comprise millions of parameters and / or weights. The step S30 of learning the domain adaptation method may comprise training of the detector, which sets the values ​​of these parameters and / or weights. Such training may typically comprise updating these values, which consists of successively correcting these values ​​according to the output of the detector for each input taken by the detector. During training, the input of the detector is the scene and the output comprises data (at least after appropriate training) on ​​the position of one or more (e.g. all) spatially reconfigurable objects (e.g. bounding boxes and / or annotations, as mentioned above) in this scene.

[0121] The correction of the value may be based on annotations related to each input. An annotation is a set of data associated with a particular input that allows evaluation if the output of the model is true or false. For example, as mentioned above, each spatially reconfigurable object of the scene belonging to the third domain may be encompassed by a bounding box. As mentioned above, in the context of the present disclosure, encompassing an object in a scene may mean encompassing all parts of the object that are fully visible on the scene. The bounding box may be annotated to indicate that it encompasses a spatially reconfigurable object. Thus, the correction of the value may include evaluating such annotations of the bounding box of the third domain scene input to the detector and determining, based on the evaluated annotations, whether the spatially reconfigurable object included in the bounding box is actually output by the detector. Determining whether the output of the detector includes the spatially reconfigurable object included in the bounding box may include evaluating a correspondence between the output of the detector and the spatially reconfigurable object, for example, by evaluating whether the output is true (e.g., if it includes a spatially reconfigurable object) or false (e.g., if it does not include a spatially reconfigurable object).

[0122] The above method of training a detector using annotations of a virtual dataset may be referred to as "supervised learning". The training of the detector may be performed by any supervised learning method. Once the detector is trained, the correction of values ​​stops. At this point, the detector can process new inputs (i.e., inputs unknown during the training of the detector) and return detection results. In the context of the present invention, the new inputs of the detector are the scenes of the test domain. Once trained, the detector returns two distinct outputs as the task of the "detection" means jointly performing the tasks of recognition (or classification) and localization of spatially reconfigurable objects in the real scenes of the test dataset.

[0123] The task of localization involves computing bounding boxes that each encompass a spatially reconstructable object of the real scene input to the detector. As mentioned before, a bounding box is a rectangular box whose axes are parallel to the sides of the image and is characterized by four coordinates. In the example, the detector returns for each spatially reconstructable object of the scene input to the detector a bounding box centered on the object at the appropriate ratio and scale.

[0124] The classification task involves labeling each computed bounding box with a corresponding label (or annotation) and associating a confidence score with the label. The confidence score reflects the detector's confidence that the bounding box truly contains a spatially reconstructable object of the scene provided as input. The confidence score may be a real number between 0 and 1. In such a case, the closer the confidence score is to 1, the more confident the detector is that the label associated with the corresponding bounding box truly annotates a spatially reconstructable object.

[0125] An example of step S30 of training a domain adaptive neural network will now be described with reference to FIG. 3, which shows a flow chart illustrating an example of step S30 of training a domain adaptive neural network.

[0126] Referring to the flowchart of FIG. 3, the step S30 of training the domain adaptation neural network may include a step S300 of providing a teacher extractor. The teacher extractor is a machine learning neural network configured to output an image representation of a real scene. In such a case, the step S30 of training the domain adaptation neural network further includes training a student extractor S310. The student extractor is a neural network configured to output an image representation of a scene belonging to a third domain. The step S310 of training the student extractor includes minimizing a loss. For each scene of the one or more real scenes, the loss penalizes the disparity between the results of applying the teacher extractor to the scene and the results of applying the student extractor to the scene.

[0127] The student extractor is a neural network configured to output an image representation of a scene belonging to a third domain. The student extractor may be part of a detector, and thus the student extractor may be further configured to output said image representation of the scene belonging to the third domain to other parts of the detector. At least one of such other parts may be configured for a task of localization, and at least one of such other parts may be configured for a task of classification, as described above. Thus, the student extractor may output an image representation of the scene in the third domain to said other parts, enabling said other parts to perform the tasks of localization and classification. Alternatively, the student extractor may be part of a domain adaptation neural network, in which case the image representation may be output to other parts (such as a detector) of the domain adaptation neural network.

[0128] This means that the student extractor monitors the output of image representations of scenes belonging to the third domain, while at least one other part of the domain adaptation neural network (e.g. the aforementioned detector) monitors the detection (e.g. localization and classification) of spatially reconstructable objects in each scene of the third domain. Said detection can be based on said image representations output by the student extractor. In other words, the image representations output by the student extractor are fed to the remaining part of the domain adaptation neural network, for example the remaining part of the detector in case the student extractor forms part of the detector.

[0129] The student extractor is trained with the help of the teacher extractor provided (S300). The teacher extractor is a machine learning neural network configured to output an image representation of a real scene. In an embodiment, there is a means that the teacher extractor is trained on a real scene, for example by learning a robust convolution filter for the real data style, to output an image representation of the real scene. A layer of neurons of the teacher extractor typically encodes a representation of the real image to be output. The teacher extractor may be trained on any dataset of real scenes available, for example from an open source, and by any machine learning technique. The step S300 of providing the teacher extractor may include a step of accessing a database in which the teacher extractor is stored after learning, and a step of retrieving the teacher extractor from the database. It should be understood that the teacher extractor may already be available, i.e. the domain adaptation learning method may not include the training of the teacher extractor only by the step S300 of providing an already available teacher extractor. Alternatively, the domain adaptation learning method may include training a teacher extractor by any machine learning technique prior to step S300 of providing a teacher extractor.

[0130] With further reference to the flowchart of FIG. 2, the step S320 of training the student extractors will now be described.

[0131] The training step S320 comprises, for each scene of one or more real scenes, minimizing a loss that penalizes a mismatch between the result of applying a teacher extractor to the scene and the result of applying a student extractor to the scene. At least some (e.g. all) of the one or more real scenes may be scenes of a test dataset and / or at least some (e.g. all) of the one or more real scenes may be scenes of another dataset of real scenes belonging to a second domain, e.g. not annotated and not necessarily numerous. The minimization of the loss comprises the results of applying both the teacher extractor and the student extractor to each of the one or more real scenes, which means that a real scene is given to both the teacher extractor and the student extractor during the step S310 of training the latter. Thus, at least a part of the domain-adaptive neural network, i.e. the student extractor, is trained on real data, while at least a part of the domain-adaptive neural network is trained on virtual data belonging to a third domain, such as the aforementioned detectors.

[0132] The loss may be a quantity (e.g., a function) that measures, for each real scene of the one or more real scenes, the similarity and / or dissimilarity between the result of applying the teacher extractor to the scene and the result of applying the student extractor to the scene. The loss may, for example, be a function that takes each of the one or more real scenes as a parameter, taking as input the result of applying the teacher extractor to the scene and the result of applying the student extractor to the scene, and outputting a quantity (e.g., a positive real number) that represents the similarity and / or dissimilarity between the result of applying the teacher extractor to the scene and the result of applying the student extractor to the scene. For each scene of the one or more real scenes, the difference between the result of applying the teacher extractor to the scene and the result of applying the student extractor to the scene is a quantification of the difference between the result of applying the teacher extractor to the scene and the result of applying the student extractor to the scene. Penalizing the imbalance may mean that the loss is an increasing function of the imbalance.

[0133] Minimizing the loss by penalizing the discrepancy between the outputs of the teacher extractor and the student extractor improves the robustness of the step S30 of learning the domain adaptation neural network. In particular, during the step S320 of training the student extractor, the teacher extractor, which has already been learned and therefore does not change its weights and / or parameters, guides the student extractor. In fact, since the discrepancy between the output of the student extractor and the output of the teacher extractor is penalized, the parameters and / or weights of the student extractor are modified so that the student extractor learns to output real image representations that are close to those output by the teacher extractor in terms of the data distribution. In other words, the student extractor is trained to imitate the teacher extractor in the way that it outputs real image representations. Thus, the student extractor can learn the same or substantially the same real data style robust convolution filters as those learned by the teacher extractor during the training of the teacher extractor. As a result, the student extractor outputs real image representations of scenes originally belonging to the third domain, even if these scenes are relatively distant from the second domain (consisting of real scenes) in terms of data distribution. This allows the domain adaptation neural network, or its detector, where appropriate, to be invariant to data appearance, since the way in which the student extractor is trained (S320) allows the domain adaptation neural network to infer spatially reconstructable objects from the real scene while being trained on annotated virtual scenes of the third domain.

[0134] The loss may be referred to as the "distilled loss" and the step S320 of training the student extractor using the teacher extractor may be referred to as the "distillation approach". Minimizing the distilled loss may be performed by any minimization algorithm or any relaxation algorithm. The distilled loss may be part of the loss of the domain adaptive neural network, optionally including the loss of the detector as mentioned above. In one embodiment, the step S30 of training the domain adaptive neural network may include minimizing the following types of losses of the domain adaptive neural network:

number

number

number

number

number

number

number

[0135] In an example, the result of applying a teacher extractor to a scene is a first Gram matrix, and the result of applying a student extractor to a scene is a second Gram matrix. The Gram matrix of an extractor (e.g., student or teacher) is a representation of the distribution of data over the image output by the extractor. More specifically, applying the extractor to a scene generates an image representation, and computing the Gram matrix of the image representation generates an embedding of the distribution of the image representation, a so-called Gram-based representation of the image. This means in an example that the result of applying a teacher extractor to a scene is a first Gram-based representation of the image representation output by the teacher extractor resulting from computing a first Gram matrix of the image representation output by the teacher extractor, and the result of applying a student extractor to a scene is a second Gram-based representation of the image representation output by the student extractor resulting from computing a second Gram matrix of the image representation output by the student extractor. For example, the Gram matrix may include non-localized information about the image output by the extractor, such as texture, shape, and weights.

[0136] In examples, a first Gram matrix is ​​calculated on several layers of neurons of the teacher extractor, the layers of neurons including at least the last layer of neurons. In these examples, a second Gram matrix is ​​calculated on several layers of neurons of the student extractor, the layers of neurons including at least the last layer of neurons.

[0137] The last layer of neurons of the extractor (teacher or student) encodes the image output by the extractor. Both the first and second Gram matrices are computed on several layers of neurons, a deeper layer of neurons. The means is that both contain more information about the output image than the image itself. This makes the training of the student extractor more robust, since it increases the level of real-style closeness between the outputs of the teacher and student extractor, which is what the loss minimization aims at. In particular, the student extractor is thereby able to accurately infer a two-layer representation of the real image, as well as its teacher.

[0138] In the example, mismatch is the Euclidean distance between the result of applying the teacher extractor to the scene and the result of applying the student extractor to the scene.

[0139] Here, we will describe the implementation of the minimization of distillation losses. In this embodiment,

number

number

number

number

number

number

number

number

[0140] The intuition behind the proposed distillation approach is to ensure that during training, the weights of the student extractors are updated such that they output a data-style representation that is close to what the teacher extractor produces on real data. To do so, the training step S30 involves training a Gram-based distillation loss at the feature extractor level.

number

number

number

number

number

number

number

number

number

[0141] As mentioned above, the step S30 of training the domain-adaptive neural network may include minimizing the following types of losses:

number

[0142] Returning to the flowchart of FIG. 1, step S50 for determining the intermediate domain will now be described.

[0143] The step S50 of determining the intermediate domain is such that, in terms of data distribution, the intermediate domain is closer to the training domain than the test domain. Thus, the step S50 of determining the intermediate domain specifies for each scene of the test dataset of scenes an inference of another scene, so that all of the inferred other scenes form a domain closer to the training domain than the test domain in terms of data distribution, i.e. the intermediate domain. The inference of said other scene may include a transformation (e.g., conversion) of said scene to said other scene. The transformation may for example consist of transforming a scene to said other scene using a generator capable of transferring images from one domain to another. It should be understood that such a transformation concerns only one or more parts of a scene, said one or more parts being transformed, for example using a generator, while the remaining part of the scene remains unchanged. The said other scene may be composed of a scene in which one or more parts are transformed and the remaining part of the scene is left unchanged. Alternatively, the transformation may include replacing the remaining part of the scene with a part of another scene (e.g. belonging to the training domain). In such a case, said other scene may be in a scene in which one or more parts are transformed and the remaining part of the scene is replaced by said part of some other scene. It should be appreciated that inference may be performed similarly for all scenes in the test domain, e.g., using the same paradigm for all scenes.

[0144] The domain adaptation neural network is trained not on the training domain, but on data obtained from the training domain. Such data may be the result of processing the data of the training domain. For example, the data obtained from the training domain may be a dataset of scenes obtained by transforming the respective scenes of the training domain. This means that the data obtained from the training domain may form a dataset of scenes belonging to a domain that is the true training / learning domain of the domain adaptation neural network. Such a domain may be determined exactly as a third domain, i.e. by performing similar operations and steps. In any case, the step S50 of determining the intermediate domain may be performed (for example through transformation of the scene as described above) such that the intermediate domain is closer to the domain formed by the data obtained from the training domain than the test domain. In an example, the data obtained from the training domain forms another intermediate domain that is closer to the intermediate domain in terms of data distribution than the training domain. Said another intermediate domain may be the third domain as described above. In all these examples, the step S50 of determining the intermediate domain corresponds to a pre-processing of the test dataset so as to bring it closer to the true training / learning set on which the domain adaptation neural network was trained. As described above, this improves the output quality of the domain adaptation neural network.

[0145] In an example, determining the intermediate domain S50 involves, for each scene in the test data set, transforming the scene in the test data set to another scene that is closer to the training domain in terms of data distribution.

[0146] Transforming a scene into another scene may include using a generator capable of transferring images from one domain to another. It should be understood that such transformation concerns only one or more parts of a scene, said one or more parts being transformed, for example, using a generator, while the remaining parts of the scene remain unchanged. The other scene may be composed of a scene where one or more parts are transformed and the remaining parts of the scene are not changed. Alternatively, the transformation may include replacing the remaining parts of the scene with parts of another scene (e.g., belonging to the training domain). In such a case, the other scene may be in a scene where one or more parts are transformed and the remaining parts of the scene are replaced by said parts of some other scene. It should be understood that inference may be performed similarly for all scenes of the test domain, for example, using the same paradigm for all scenes.

[0147] An example of step S50 for determining the intermediate domain will now be described with reference to FIG. 4, which shows a flow chart illustrating an example of step S50 for determining the intermediate domain.

[0148] 4, the transformation of the scene may include a step S500 of generating a virtual scene from a scene of the test dataset. In such a case, another scene is inferred based on the scene of the test dataset and the generated virtual scene.

[0149] The generating step S500 can be performed by applying any generator capable of generating a virtual scene from a scene of a test domain. Another scene may be inferred by blending a scene of a test dataset with the generated virtual scene and / or by combining (e.g., blending, seamless cloning, or merging, e.g., by image channel merging methods) a scene of a test dataset with the generated virtual scene.

[0150] The virtual scene generator takes as input a scene of a test domain and outputs a virtual scene, e.g., a virtual scene that is less photorealistic than the scene of the test domain. It should be understood that the virtual generator may already be trained / learned. Thus, the step S500 of generating a virtual scene may include providing the virtual scene generator, e.g., by accessing a memory in which the virtual scene generator is stored after training / learning of the virtual scene generator and retrieving the virtual scene generator from the memory. Additionally or alternatively, the provision of the virtual scene generator may be preceded by learning / training of the virtual scene generator by any machine learning technique suitable for such training.

[0151] The virtual scene generator takes as input a scene of a test domain and outputs a virtual scene, e.g., a virtual scene that is less photorealistic than the scene of the test domain. It should be understood that the virtual generator may already be trained / learned. Thus, generating the virtual scene S500 may include providing the virtual scene generator, e.g., by accessing a memory in which the virtual scene generator is stored after training / learning of the virtual scene generator and retrieving the virtual scene generator from the memory. Additionally or alternatively, providing the virtual scene generator may be preceded by learning / training of the virtual scene generator by any machine learning technique suitable for such training.

[0152] In examples, the virtual scene generators have been trained on a dataset of scenes each including one or more spatially reconfigurable objects. In such examples, when the virtual scene generator is fed a scene including one or more spatially reconfigurable objects, it may output a virtualization of the scene, which is a virtual scene that also includes one or more spatially reconfigurable objects, e.g., objects all in the same position and / or configuration. As discussed above, it should be understood that the domain adaptation inference method and / or process may include only providing a virtual scene generator (i.e., without training thereof) or may further include training the virtual scene generator.

[0153] In an implementation of the domain adaptation inference method and / or process, a test data set is created from a real manufacturing scene including a manufacturing robot, and the generator is a CycleGAN real-virtual network trained on two data sets of manufacturing scenes including the manufacturing robot, a virtual data set and a real data set. The method and / or process includes in this embodiment training a CycleGAN network on the two data sets to adapt the appearance of the robot in the test images of the test scene to approximate the training set of the domain adaptation neural network, in this embodiment a third domain by the domain adaptation learning method, created from a virtual manufacturing scene including a virtual manufacturing robot. The training results in a real-virtual robot generator that can modify the real test images to virtualize the real test images.

[0154] FIG. 22 shows an example of the generating step S500 using the CycleGAN generator. In this example, a test dataset is created from a real manufacturing scene, each including one or more manufacturing robots. FIG. 22 shows a first real scene 220 and a second real scene 222 of the test dataset. Both scenes are manufacturing scenes including articulated robots. FIG. 22 further shows a first virtual scene 224 and a second virtual scene 226 generated from the first scene 220 and the second scene 222, respectively, by applying the CycleGAN generator.

[0155] 4, the step S50 of determining the intermediate domain may include a step S510 of blending a scene of the test dataset with the generated virtual scene, in which case the blending may result in another scene.

[0156] Formally,

number

number

number

number

number

number

number

number

[0157] An example of linear blending is shown in Figure 23. Figure 23 shows a real manufacturing scene 230 including a real manufacturing robot and a virtual scene 232 generated from the real scene 230. Figure 23 further shows the result of linear blending, which is a virtual manufacturing scene 234.

[0158] First, the step S60 of inferring spatially reconfigurable objects from a scene may include the steps of inferring a spatially reconfigurable object from one scene and, for each scene of an input data set of scenes, inferring a spatially reconfigurable object from each scene of the input data set of scenes. In particular, the inferring step S60 may constitute a testing phase of a domain adaptive neural network, where all scenes of the test data set are provided (e.g., successively) as input to a domain adaptive neural network that infers one or more spatially reconfigurable objects in each input scene. Additionally or alternatively, the inferring step S60 may constitute an application phase of a domain adaptive learning method and / or a domain adaptive inference method and / or process, where the input data set is a data set of scenes of a test domain that is not equal to the test data set. In such a case, the application phase may include providing an input data set, inputting each scene of the input data set to the domain adaptive neural network, and performing inference (S60) for each scene. It should also be understood that step S60 of inferring spatially reconfigurable objects in a scene may include steps of inferring (e.g., simultaneously or consecutively) one or more spatially reconfigurable objects in a scene that includes one or more of such objects, for example by inferring all spatially reconfigurable objects in the scene.

[0159] First, the step S60 of inferring spatially reconfigurable objects from a scene may include the steps of inferring a spatially reconfigurable object from one scene and, for each scene of an input data set of scenes, inferring a spatially reconfigurable object from each scene of the input data set of scenes. In particular, the inference S60 may constitute a testing phase of a domain adaptive neural network, where all scenes of the test data set are provided (e.g., successively) as input to a domain adaptive neural network that infers one or more spatially reconfigurable objects in each input scene. Additionally or alternatively, the inference step S60 may constitute an application phase of a domain adaptive learning method and / or a domain adaptive inference method and / or process, where the input data set is a data set of scenes of a test domain that is not equal to the test data set. In such a case, the application phase may include providing an input data set, inputting each scene of the input data set to the domain adaptive neural network, and performing inference (S60) for each scene. It should also be understood that step S60 of inferring spatially reconfigurable objects in a scene may include steps of inferring (e.g., simultaneously or consecutively) one or more spatially reconfigurable objects in a scene that includes one or more of such objects, for example by inferring all spatially reconfigurable objects in the scene.

[0160] We now describe the step S60 of estimating one spatially reconfigurable object in one scene, which may be part of the step S60 of estimating more spatially reconfigurable objects in the scene. The step S60 of inferring spatially reconfigurable objects in a scene by applying a domain adaptive neural network consists in inputting the scene to the domain adaptive neural network. The domain adapted neural network then outputs data on the positions of the spatially reconfigurable objects. For example, the neural network may output a (e.g. digital) image representation of the scene, each bounding box containing the inferred spatially reconfigurable object. The inference may also include the calculation of such bounding boxes and / or the labeling of the bounding boxes (e.g. as containing the spatially reconfigurable object of the scene) and / or the labeling of the inferred spatially reconfigurable object (as a spatially reconfigurable object). The output of the neural network may be stored in a database and / or displayed on a computer screen.

[0161] Figures 24-29 show the inference of manufacturing robots in real manufacturing scenes. Figures 24-29 respectively show the corresponding real scenes of the test domain fed as input to the domain adaptation neural network and the bounding boxes computed and annotated around all manufacturing robots included in the corresponding scenes. As shown in Figures 26-30, in the real scenes, some robots and / or robots that are at least partially occluded can be inferred.

[0162] The inference of spatially reconfigurable objects may in fact be performed by a detector forming at least part of a domain-adaptive neural network that jointly performs the tasks of recognition (or classification) and localization of spatially reconfigurable objects in real scenes of a test dataset. The localization task involves computing bounding boxes that each encompass a spatially reconfigurable object of the scene input to the detector. As mentioned before, a bounding box is a rectangular box whose axes are parallel to the sides of the image and is characterized by four coordinates. In an example, the detector returns for each spatially reconfigurable object of the scene input to the detector a bounding box centered on the object at an appropriate ratio and scale. The classification task involves labeling each computed bounding box with a corresponding label (or annotation) and associating a confidence score to the label. The confidence score reflects the confidence of the detector that the bounding box truly encompasses the spatially reconfigurable object of the scene provided as input. The confidence score may be a real number between 0 and 1. In such cases, the closer the confidence score is to 1, the more confident the detector is that the label associated with the corresponding bounding box truly annotates a spatially reconstructible object.

[0163] In an example where the domain adaptation neural network is configured for the inference of objects in real scenes, for example when the test domain is made from a real scene and the intermediate domain is made from a virtual scene, the domain adaptation neural network may further comprise an extractor. The extractor outputs image representations of the real scenes and feeds them to other parts of the domain adaptation neural network, for example a detector or part of a detector, forming part of the domain adaptation neural network configured for example to monitor the detection of spatially reconstructable objects. In an example, this means that the extractor is able to process the scenes of the intermediate domain to output image representations of the scenes, for example by learning robust convolution filters for real data style. A layer of neurons of the extractor typically encodes a representation of the real or photorealistic image to be output. This means that the extractor monitors the output of image representations of scenes belonging to the intermediate domain, while at least one other part of the domain adaptation neural network (for example the aforementioned detector) monitors the detection (for example localization and classification) of spatially reconstructable objects in each input scene of a third domain. The detection can be based on the image representations output by a student extractor, allowing for efficient processing and output of real data.

[0164] Each scene provided as an input to the domain adaptation neural network belongs to the test domain, but is a scene transferred on the determined intermediate domain. In other words, providing a scene as an input to the domain adaptation neural network includes transferring the scene on the intermediate domain and feeding the transferred scene as an input to the domain adaptation neural network. The scene transfer can be performed by any method that can transfer a scene from one domain to another. In an example, the scene transfer can include transforming the scene into another scene and, optionally, blending the scene with the transformed scene. In such a case, the transformation and optionally blending can be performed as in the example described above with reference to the flowchart of FIG. 4. In particular, the scenes of the intermediate domain may actually be scenes of the test dataset that have been transferred on the intermediate domain.

[0165] Next, the process implementation will be described.

[0166] This implementation proposes a learning framework in a virtual world to address the manufacturing data scarcity problem. In particular, this implementation utilizes a virtual simulation dataset for learning and the efficiency of image processing methods, namely deep neural networks (DNNs), to train a domain-adaptive neural network.

[0167] Virtual simulation constitutes one form of obtaining virtual data for training. The latter has emerged as a promising technique to address the difficulty of obtaining the training data required for machine learning tasks. Virtual data is algorithmically generated to mimic the characteristics of real data, but with automatic, error-free, and cost-free labeling.

[0168] In order to better explore the working mechanisms of different manufacturing equipment, a virtual simulation is proposed with this implementation. Formally, different manufacturing tools are taken into account in the learning process via the different states they can present when they work.

[0169] Deep Neural Networks: A powerful set of techniques for learning in neural networks

[19] , which is a biologically inspired programming paradigm that allows computers to learn from observed data. In image recognition, the success of DNNs is attributed to their ability to learn rich mid-level media representations, as opposed to the hand-engineered low-level features (Zernike moments, HOG, Bag-of-Words, SIFT, etc.) used in other image classification methods (SVM, Boosting, Random Forest, etc.). More specifically, DNNs focus on end-to-end learning based on raw data. In other words, they move away from feature engineering to the maximum extent possible by achieving end-to-end optimization starting with raw features and ending with labels.

[0170] Domain Adaptation: A field related to machine learning and transfer learning. This scenario arises when we aim to learn a model from a source data distribution that performs well on a different (but related) target data distribution. The goal of domain adaptation is to find an effective mechanism to transfer or adapt knowledge from one domain to another. This is achieved by introducing a cross-domain bridging component to improve the cross-domain gap between the train and test data.

[0171] The proposed implementation is particularly illustrated by FIG. 31, which shows the offline and online stages of the process.

[0172] 1. Offline Phase: This phase aims to train the model using virtual data that is considered competent in reasoning in the real world. It includes two main steps. Note that this phase is transparent to the user.

[0173] (1) Generate representative virtual training data that is automatically annotated through virtual simulation in a virtual manufacturing environment that is geometrically realistic but not photorealistic. The type of annotation depends on the target inference task.

[0174] (2) A neural network model is trained based on the virtual data, which consists of a domain-adapted DNN-based model.

[0175] 2. Online Phase: Given real-world media, the learned domain adaptation model is applied to infer manufacturing devices.

[0176] State-of-the-art object detectors are based on deep learning models. Millions of parameters, whose values ​​cannot be set manually, characterize these models. These parameters must therefore be set thanks to a learning algorithm. When the learning algorithm is updating the model parameters, the model is said to be in "training mode". It consists in continuously "correcting" the model for each input, thanks to the annotations associated with each input. An annotation is a set of data associated with a particular input that allows to evaluate whether the output of the model is true or false. For example, an object classifier trained to distinguish between images of cats and dogs needs a dataset of annotated images of cats and dogs, each annotation being "cat" or "dog". Thus, in its training mode, if the object classifier outputs "dog" for a cat image in the input, the learning algorithm corrects the model by updating its parameters. This method of supervising the training of a model thanks to an annotated dataset is called "supervised learning". Once the model is trained, we stop updating its parameters. The model is then only used to process new inputs (i.e. unknown inputs during the training mode) and return detection results. This is called "test mode".

[0177] State-of-the-art object detectors are based on deep learning models. Millions of parameters, whose values ​​cannot be set manually, characterize these models. These parameters must therefore be set thanks to a learning algorithm. When the learning algorithm is updating the model parameters, the model is said to be in "training mode". It consists in continuously "correcting" the model for each input, thanks to the annotations associated with each input. An annotation is a set of data associated with a particular input that allows to evaluate whether the model's output is true or false. For example, an object classifier trained to distinguish between images of cats and dogs needs a dataset of annotated images of cats and dogs, each annotation being "cat" or "dog". Thus, in its training mode, if the object classifier outputs "dog" for a cat image in the input, the learning algorithm corrects the model by updating its parameters. This method of supervising the training of the model thanks to annotated datasets is called "supervised learning". Once the model has been trained, it stops updating its parameters. After that, the model is only used to process new inputs (i.e. unknown inputs during training mode), returning detection results is called "test mode".

[0178] In the context of the present embodiment, the object detector returns two distinct outputs for the task of "detection" to a means that jointly performs the tasks of recognition (or classification) and localization.

[0179] 1. Localization Output: Object localization is done thanks to bounding boxes. A bounding box is a rectangular box whose axes are parallel to the sides of the image. It is characterized by four coordinates. Ideally, the object detector returns for each object a bounding box at the object's center, with the right ratio and scale.

[0180] 2. Classification Output: Object classification is done thanks to the category label associated with each bounding box's confidence score. The confidence score is a reality between 0 and 1. The closer it is to 1, the more confident the object detector is about the category label associated with the corresponding bounding box.

[0181] This implementation is particularly illustrated by FIG. 32, which shows the domain adaptation components used in both the offline and online phases.

[0182] The present invention relates to training object detection models using generated virtual data. To address the virtual-real domain gap in both the offline and online stages, we applied several domain adaptation approaches.

[0183] 1. Offline Phase: This phase aims to train the model using virtual data that is considered competent in making inferences in the real world. This phase includes two main steps.

[0184] (1) Virtual simulation in a virtual manufacturing environment generates representative virtual training data that is automatically annotated with bounding boxes and “robot” labels.

[0185] (2) Domain Adaptation Transformation (i) At the training set level, the latter is transformed into a new training domain that is closer to the real world in terms of data distribution. (ii) At the model architecture level, a domain adaptation module is inserted into the model to address the problematic virtual-to-real domain shift.

[0186] 2. Online Phase: Real-world media is transformed into a new test domain that is better handled by the learned detection model and then inferred by the latter. In other words, the new test domain is closer to the training domain than the original real domain.

[0187] It is important to note that one peculiarity of this embodiment is that it does not encounter the challenge of virtual-real image conversion accuracy. The intuition behind the current work is to consider an intermediate domain between the virtual and real worlds, instead of trying to transfer the virtual data to the real world. Formally, the virtual training data and real test data are dragged to the training and test intermediate domains, respectively, where the domain shift is minimal.

[0188] In the light of the above framework, it can be seen that the implementation currently discussed is based on two main steps. · Virtual data generation for learning. Domain Adaptation.

[0189] To address the scarcity of factory images, this implementation is based on a simulation-based virtual data generation approach. The training set consists of 3D rendered factory scenes. These factory scenes are generated by the company's internal design application with masks that segment the objects of interest (robots) at pixel level. Each scene is simulated into multiple frames that describe multiple functional states that the robot may have.

[0190] This step generates a set of virtual images with distinctive and complex features for the task of detecting robots in manufacturing situations. Although it seems like a "simple" single object detection task, baseline detectors and state-of-the-art domain adaptation networks are not capable in this problem. Formally, the generated set is not photorealistic but rather "cartoonish". The simplicity of the generated virtual scenes reflects the ease of generating our training data, which is the main advantage of this embodiment. At the same time, the complexity of the objects of interest makes it difficult to envision using state-of-the-art approaches to tackle this kind of virtual data, namely domain randomization DR techniques (see [7, 15]).

[0191] In fact, domain randomization abandons photorealism by randomly perturbing the environment in a non-photorealistic way. The intuition behind this DR is to teach the algorithm to learn to focus on the essential features of the image and ignore objects in the scene that are not of interest. More formally, the main realistic features that are missing in the virtual data are the low-level morphology of the image, e.g., texture, lighting, and shading. DR ensures the perturbation of these features across the training dataset so that the algorithm does not rely on them to recognize objects. Following the same logic, DR does not affect common features between real and virtual data. In the scope of the virtual data generation used, it can be seen that the correct features that the learning model is expected to rely on to learn are shape and contour. This is important in DR techniques.

[0192] Moreover, the objects of the present embodiment, which are robots, have many joint configurations (typically 6 degrees of freedom) that are calibrated by a range of continuous values. This gives each robot type an infinite range of possible states. Therefore, randomizing all features except shape and contour for learning is not optimal for this kind of complex industrial object. In practice, the variability required to represent the shape and contour of the object is difficult to meet when the object is complex. In particular, in the robot example, it is practically inconceivable to learn a model that can extract and focus on the characteristic shapes and contours of various robot types with infinite joint positions in order to recognize said object in an image.

[0193] The proposed implementation constructs a competent learning problem when the data is complex, as in this case.

[0194] The proposed domain adaptation approach can be thought of as a partial domain randomization technique. Formally, we propose to augment the correct features that the model is learning by randomizing the data features or transforming them into a new domain that better represents the real world than the virtual one. For example, we can better explore idiosyncrasies in texture properties. To do so, three novel techniques are implemented in the currently discussed implementation:

[0195] 1. Virtual to real conversion To improve cross-domain shift, a common approach in state-of-the-art papers is to modify the appearance of the virtual data so that it looks as if it was drawn from the real domain, i.e., the virtual data acquires a distribution that better represents the real world. To do so, the virtual image is modified as a whole using the real scene, without any correspondence between similar objects. However, it can be speculated that to obtain optimal performance of image-to-image translation, it would be beneficial to transfer the real robot texture to the robot in the virtual scene and the same for the background. The current implementation builds on this targeted appearance transfer by applying different image-to-image transformation networks separately to the robot and factory backgrounds in each medium from the training set.

[0196] To this end, we use the CycleGAN model

[20] to translate images from one domain to another. We propose a method that can capture special features of one set of images and understand how to translate these features to a second collection of images, without the need for full paired training examples. This relaxation of having a one-to-one mapping makes this formulation very powerful in practice. This is achieved by a generative model, specifically a generative adversarial network (GAN) called CycleGAN.

[0197] As mentioned above, the two different models are trained separately. CycleGAN Real-Real Network 1: Trained on two factory background sets, the real set and the real set. The result is a virtual-real factory generator that is used to retouch the background in a virtual image. CycleGAN Virtual-Real 2: Trained on two sets of robots, a virtual set and a real set. The result is a virtual-real robot generator that is used to modify robots in a virtual image.

[0198] 2. Gram-based distillation To better highlight the robustness of our model with respect to domain shifts, we propose a novel knowledge distillation approach implemented at the detector level. Note that most state-of-the-art detectors (see [18, 9]) contain a feature extractor, which is a deep network of convolutional layers. The feature extractor outputs an image representation to be fed into the rest of the detector. The intuition behind our distillation is to guide the detection model to learn robust convolution filters for the real data style, which is consistent with our main objective of making the model invariant to the data appearance.

[0199] Formally, our implementation consists of training a detector on a virtual medium and simultaneously updating the weights of the detector's feature extractors to approximate the style encoded by a two-layer representation of a real image that is fed into both our detector and a second detector pre-trained on real data from an open source.

[0200] A certain layer

number

number

number

number

number

number

number

number

[0201] The intuition behind the proposed distillation is that during training, the weights of the detector's feature extractors are updated to ensure that the detector outputs a data-style representation that is close to what feature extractors pre-trained on real data would produce. To do so, the currently discussed implementation uses a Gram-based distillation loss at the feature extractor level.

number

[0202] Actual image (this

number

number

number

number

number

number

number

number

[0203] 3. Real to Virtual Conversion It can be expected that the task of transforming a virtual image into the real domain cannot be accomplished perfectly. The means is that the modified data is closer to the real world than the virtual data, without a perfect match to the real distribution. Therefore, the modified virtual data is a new domain that lies between the synthetic and real domains.

number

number

number

number

[0204] In the context of the present embodiment, the transformation is ensured by training a CycleGAN real-virtual network trained on two robot sets: a virtual set and a real set. This results in a real-virtual robot generator that is used to modify real test images. At this level, we chose to train the CycleGAN network on the robot set, since we are primarily interested in adapting the appearance of the robots in the test images to be recognized by the detector. It should also be noted that it is also possible to perform more advanced image transformations on the test data by targeting the domains separately, as was done for the training set. However, the improvement is limited and we are content to apply one CycleGAN network to the entire test images.

Claims

1. 1. A computer-implemented method of machine learning, comprising: Providing (S10) a dataset of virtual scenes, a dataset of virtual scenes belonging to a first domain, and a test dataset of real scenes belonging to a second domain; determining (S20) a third domain, the third domain being closer to the second domain than the first domain in terms of data distribution; training (S30) a domain-adaptive neural network based on a third domain, the domain-adaptive neural network being a neural network configured for inference of spatially reconfigurable objects in real scenes; having The step of determining the third domain (S20) comprises, for each scene of the data set of virtual scenes: Extracting one or more spatially reconstructable objects from a scene (S200); For each extracted object, transforming the extracted object into an object closer to the second domain in terms of data distribution than the extracted object (S210). Including, The step of determining the third domain (S20) further comprises: For each transformed extracted object, placing the transformed extracted object in one or more scenes that are closer to the second domain than to the first domain in terms of data distribution (S220). Including, The third domain includes each scene in which the transformed extracted object is placed. Computer-implemented method.

2. placing the transformed extracted object in the one or more scenes (S22); 0) is executed randomly The method of claim 1.

3. The step of determining the third domain (S20) includes a step of placing one or more distractors in a respective one of the one or more scenes included in the third domain (S230), the distractors being objects that are not spatially reconfigurable objects. The method according to claim 1 or 2.

4. The step of training the domain adaptive neural network (S30) includes: providing a teacher extractor (S300), the teacher extractor being a machine learning neural network configured to output an image representation of a real scene; Training a student extractor (S310), the student extractor being a neural network configured to output an image representation of a scene belonging to a third domain, the step of training the student extractor (S310) comprising, for each scene of the one or more real scenes, minimizing a loss that penalizes a discrepancy between a result of applying the teacher extractor to the scene and a result of applying the student extractor to the scene.

4. The method according to claim 1 .

5. A result of applying the teacher extractor to the scene is a first Gram matrix, and a result of applying the student extractor to the scene is a second Gram matrix. The method according to claim 4.

6. the first Gram matrix is ​​computed on several layers of neurons of the teacher extractor, said layers of neurons including at least a last layer of neurons; The second Gram matrix is ​​computed on several layers of neurons of the student extractor, the layers of neurons including at least the last layer of neurons. The method according to claim 5.

7. The discrepancy is the Euclidean distance between the results of applying the teacher extractor to the scene and the results of applying the student extractor to the scene.

7. The method according to any one of claims 4 to 6.

8. Each virtual scene in the dataset of virtual scenes is a virtual manufacturing scene including one or more spatially reconfigurable manufacturing tools, and the domain-adaptive neural network is configured for inference of the spatially reconfigurable manufacturing tools in a real manufacturing scene.

8. The method according to any one of claims 1 to 7.

9. Each real scene in the test dataset is a real manufacturing scene including one or more spatially reconfigurable manufacturing tools, and the domain-adaptive neural network is configured for inference on the spatially reconfigurable manufacturing tools in the real manufacturing scenes.

9. The method according to any one of claims 1 to 8.

10. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 9.

11. An apparatus including a data storage medium having the computer program of claim 10 recorded thereon.

12. The apparatus of claim 11 , further comprising a processor coupled to the data storage medium.

Images