CROSS-DOMAIN DISTANCE LEARNING SYSTEM AND METHOD

Abstract

To provide an augmented reality (AR) system and method that may include a controller operable to process one or more convolutional neural networks (CNN) and a visualization device operable to acquire one or more 2-D RGB images.SOLUTION: The controller may generate an anchor vector in a semantic space in response to an anchor image provided to a first convolutional neural network (CNN). The anchor image may be one of the 2-D RGB images. The controller may generate a positive vector and a negative vector in the semantic space in response to a negative image and a positive image being provided to a second CNN. The negative and positive images may be provided as 3-D CAD images. The controller may apply a cross-domain deep metric learning algorithm that is operable to extract image features in the semantic space using the anchor vector, positive vector and negative vector.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The following relates generally to machine learning augmented reality (AR) systems and methods. [Background technology]

[0002] 2. Background of the Invention In general, machine learning (ML) algorithms are operable to perform a given task using patterns and inference. Generally, machine learning algorithms are based on mathematical models that rely on "training data" to make predictions or decisions without being explicitly programmed to perform a required task. Summary of the Invention [Means for solving the problem]

[0003] Summary of the Invention In one embodiment, an augmented reality system (AR system) and a method of augmented reality are disclosed that may include a controller operable to process one or more convolutional neural networks (CNNs) and a visualization device operable to acquire one or more two-dimensional RGB images. The controller may also generate an anchor vector in the semantic space in response to an anchor image provided to a first convolutional neural network (CNN). The anchor image may be one of the two-dimensional RGB images acquired by the visualization device. The controller may also generate a positive vector and a negative vector in the semantic space in response to a negative image and a positive image provided to a second CNN. The negative image may be a first three-dimensional computer-aided design (CAD) image, and the positive image may be a second three-dimensional CAD image. Both the first CAD image and the second CAD image may be provided to the AR system from a database.

[0004] The first CNN and the second CNN are one or more 1st and A second convolutional layer, one or more 1st and A second max pooling layer, 1st andA second planarization layer; 1st and Second Dropout Layer And the first and a second fully connected layer. The controller may include a cross-connector operable to extract image features in the semantic space using the anchor image, the positive image, and the negative image. domain It is also contemplated that a deep metric learning algorithm may be applied. It is further contemplated that the controller may be operable to extract one or more image features from different modalities using the anchor vectors, the positive vectors and the negative vectors.

[0005] The cross-domain deep distance learning algorithm can be implemented as a TripletLoss algorithm operable to decrease a first distance between the anchor vector and a positive vector in the semantic space and increase a second distance between the anchor vector and a negative vector in the semantic space, and the convolutional layers included in the first CNN and the second CNN can be implemented with one or more activation functions that can include a rectified linear unit.

[0006] It is contemplated that the first CNN and the second CNN may utilize a skip connection architecture. The second CNN may be designed as a Siamese network. The controller may perform the stage recognition by analyzing the extracted image features in the semantic space. Finally, the controller may also be operable to determine whether an invalid repair sequence has occurred based on the analysis of the image features in the semantic space.

[0007] It is contemplated that the controller may be further operable to determine a pose of an image object in the one or more RGB images. The controller may be adapted to apply a post-processing image algorithm to the one or more RGB images. [Brief description of the drawings]

[0008] [Figure 1] 1 is an exemplary augmented reality (AR) system according to one embodiment. [Figure 2A] FIG. 13 is an exemplary diagram illustrating the learning process of a TripletLoss function applied by a deep learning network. [Figure 2B] FIG. 13 is an exemplary diagram illustrating the learning process of a TripletLoss function applied by a deep learning network. [Diagram 3] FIG. 1 is an exemplary diagram illustrating a convolutional neural network operating during a training phase. [Figure 4] FIG. 1 is an exemplary diagram illustrating a convolutional neural network operating during a test phase or a run-time phase. [Diagram 5] FIG. 13 illustrates an RGB image processed during invalid stage detection processing. [Figure 6] FIG. 13 is a diagram showing bounding boxes for identifying parts from an RGB image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description Detailed embodiments are disclosed herein as necessary. However, it should be understood that the disclosed embodiments are merely exemplary and may be embodied in various alternative forms. The figures are not necessarily to scale, and some configurations may be exaggerated or minimized to show details of specific components. Thus, specific structural and functional details disclosed herein should not be construed as a limiting basis, but merely as a representative basis for teaching those skilled in the art the use of various aspects of the embodiments herein.

[0010] It is contemplated that a vision-based AR system operable to recognize different states or stages of a work procedure (e.g., the correct steps required to repair a vehicle) may be desired. The vision-based AR system may be operable to recognize the different stages using a three-dimensional (3D) model of the entire procedure (e.g., a 3D model of the vehicle and a subset of vehicle parts). It is contemplated that the AR system may utilize a machine learning algorithm that receives input data from different domains and encodes the input data as high-dimensional feature vectors. The machine learning algorithm may also be operable to transform the encoded features into a semantic space for distance measurement.

[0011] 1 illustrates an AR system 100 according to an example embodiment. The client system 110 may include a visualization device 112 that, when worn by a user, may be operable to capture one or more two-dimensional RGB images 113 (i.e., two-dimensional color images). For example, the visualization device 112 may include a pair of AR glasses that includes an integrated digital or video camera (e.g., a digital single-lens reflex camera or a mirrorless digital camera) operable to capture the RGB images 113.

[0012] The client system 110 may further include a controller 114 and a memory 116. The controller 114 may be one or more computing devices for processing commands, such as a quad-core processor, computer processor, microprocessor, or any other device, set of devices, or other mechanisms capable of performing the operations discussed herein. The memory 116 may be operable to store instructions and commands. The instructions may be in the form of software, firmware, computer code, or any combination thereof. The memory 116 may be one or more data storage devices of any form, such as volatile memory, non-volatile memory, electronic memory, magnetic memory, optical memory, or any other form of data storage. The memory 116 may be internal to the client system 110 (e.g., DDR memory), or the memory may include a removable memory component (e.g., microSD card memory).

[0013] RGB image 113 may be captured by visualization device 112 for further processing by controller 114. For example, controller 114 may be operable to determine the position and orientation (i.e., pose) of objects captured in RGB image 113 using a VisualOdometry algorithm, a Simultaneous Localization and Mapping (SLAM) algorithm, and / or a model-based tracking algorithm.

[0014] The client system 110 is connectable to and capable of communicating with the server system 120. For example, the client system 110 may include a client communicator 118 operable to transmit and receive data using a wired network (e.g., LAN) or using wireless network communication (e.g., Wi-Fi, cellular, Bluetooth, or Zigbee). The server system 120 may include a server communicator 122 also operable to transmit and receive data using a wired network (e.g., LAN) or using wireless network communication (e.g., Wi-Fi, cellular, Bluetooth, or Zigbee). It is contemplated that the client communicator 118 may be operable to transmit data, such as the RGB image 113 and the pose data, to the server communicator 122. It is further contemplated that the client system 110 and the server system 120 may be included in one unit, and thus, providing the client communicator 118 and the server communicator 122 may not be required.

[0015] It is contemplated that the server system 120 may also include a memory 126 similar to the memory 116 for storing the RGB image 113 and pose data received from the client system 110. The server system 120 may also include a controller 124 operable to perform post-processing algorithms on the RGB image 113, similar to the controller 114. The server system 120 may also be connected to and in communication with a database 128 that stores three-dimensional computer-aided design (3D-CAD) models (i.e., 3D images) of the target work procedure. The server system 120 may also communicate with the database 128 using a server communicator 122. It is contemplated that the database 128 may be included within the server system 120, and thus the provision of the server communicator 122 may not be required. The controller 124 may be operable to apply a computer graphic rendering algorithm to generate one or more normal map images of the 3D-CAD model, which may be compared to the RGB image 113 based on a view defined by the pose data received from the client system 110. The controller 124 may also be operable to apply post-processing image algorithms to the RGB image 113 received from the client system 110 .

[0016] The controller 124 may also be operable to apply one or more deep neural network (DNN) algorithms. It is contemplated that the controller 124 may apply a DNN state prediction algorithm that predicts the state (e.g., current stage) of a work procedure using the normal map image and the RGB image 113. The server system 120 may send instructions back to the client system 110 that may be visually displayed by the visualization device 112.

[0017] It is contemplated that the DNN may be a convolutional neural network (CNN) including convolutional and pooling layers to handle image recognition applications. The CNN may be a two-branch CNN operable to obtain bilinear feature vectors for fine-grained classification. The CNN may be operable to learn intrinsic features to compare multiple fine-grained categories of a given image. It is also contemplated that the machine learning algorithm may be operable to capture small variations in key regions of interest (ROIs) and avoid noisy background images, lighting, and viewpoint changes. It is further contemplated that the controller 124 may be operable to utilize deep metric learning (e.g., cross-domain deep metric learning) to further improve the performance of the CNN by not limiting the recognition process to only the procedures provided during the training phase.

[0018] For example, the controller 124 may be operable to utilize a CNN to determine the current step the user is performing and provide instructions for the next step the user must perform (i.e., "step recognition"). The controller 124 may also be operable to utilize a CNN to detect whether the user's actions deviate from a pre-stored sequence of steps (i.e., "invalid step detection"). Upon detecting that the user has performed an invalid step, the controller 124 may send instructions to the client system 110 to inform the user how to correct the incorrect action already taken. It is contemplated that the controller 124 may be operable to perform "step recognition" and "invalid step detection" using a 3D-CAD model stored in the database 128 that corresponds to the target work procedure being evaluated.

[0019] It is contemplated that the CNN can be designed as a cross-domain deep distance learning algorithm that is trained to determine a similarity measure between the 2D RGB images 113 acquired by the visualization device 112 and the 3D-CAD data stored in the database 128. By training the CNN, the controller 124 may be operable to compare the similarity or distance between the original data (i.e., the original 2D images and the 3D CAD model) and the data transformed into the semantic space using a variant of the TripletLoss algorithm represented by the following equation (1):

[0020]

number

[0021] Here, L triplet is the calculated TripletLoss function, and f RGB is the branch of CNN feature coding of RGB image 113, and f normal is a branch of the TripletLoss function that extracts features from normal map images stored as 3D CAD models in the database 128. It is conceivable that the TripletLoss function may operate on triplets of feature vectors including the anchor image, the positive image and the negative image.

[0022] 2A and 2B are exemplary diagrams showing how a CNN can be trained using the TripletLoss algorithm. Again, a triplet of images can be generated, including an anchor image 210, a positive image 220, and a negative image 230. FIG. 2A shows that the distance from the positive image 220 to the anchor image 210 is greater than the distance from the negative image 230 to the anchor image 210. As shown in FIG. 2B, the CNN can apply the TripletLoss function (Equation 1) to reduce the vector distance between the anchor image 210 and the positive image 220, and increase the distance between the anchor image 210 and the negative image 230.

[0023] 3 shows an anchor image 310, a positive image 320, and a negative image 330 that can be used in the TripletLoss algorithm. As shown, one of the multiple RGB images 113 acquired by the client system 110 can be assigned as the anchor image 310. The CNN can then receive the positive image 320 and the negative image 330 from a pre-stored normal map image generated by a 3D-CAD model stored in the database 128. The CNN algorithm can be trained to extract features from the anchor image 310 and the normal map image (i.e., the positive image 320 and the negative image 330) in the semantic space.

[0024] It is also contemplated that the controller 124 may be operable to execute a two-branch CNN to extract high-dimensional features from the RGB image 113 acquired by the client system 110 and the normal map image stored as a 3D-CAD model in the database 128. It is contemplated that each branch of the CNN may include one or more convolutional layers with kernels of various sizes (e.g., 3×3 or 5×5). The number and size of the kernels may be adjustable depending on a given application.

[0025] It is also contemplated that the CNN may include one or more convolutional layers that utilize rectified linear units (ReLU) or hyperbolic tangents (tanh) as activation functions, one or more normalization layers operable to improve the performance and stability of the CNN, and a max pooling layer capable of reducing dimensionality. The CNN may include one or more stacking modules based on the complexity of the distribution of the target data. The CNN may also utilize a skip connection architecture, in which the output of one layer of the CNN may not be directly input to the subsequent layer of the CNN. Alternatively, the skip connection architecture may allow the output of one layer to be associated with the input of a non-sequential layer.

[0026] Table 1 below shows an example structure that the CNN can use for one branch that may contain a design with 8,468,784 parameters. [Table 1]

[0027] As shown, the CNN may include one or more convolutional layers, one or more max pooling layers, a flattening layer, a dropout layer, and a dense layer (i.e., a fully connected layer or a linear layer). The size of the data input to each layer of the CNN (i.e., "input size") and the size of the corresponding data output by a given layer of the CNN (i.e., "output size") are shown. It is contemplated that the kernel size of each convolutional layer may be different, although it is contemplated that the kernel sizes may be equivalent depending on the given application. The CNN may also utilize a ReLU activation function for each convolutional layer. It is contemplated that the activation function is variable based on the given application, and that the CNN may utilize other known activation functions (e.g., tanh activation function).

[0028] Figure 4 illustrates a two-branch CNN 400 that may be utilized during the training phase. It is contemplated that the CNN 400 may be designed according to the architecture illustrated in Figure 4. The number of layers, the size of each layer, and the activation functions may be variable depending on a given application.

[0029] The CNN 400 may include a branch having an RGB network 410 that may receive an anchor image 420 as input data (e.g., the RGB image 113). The RGB network 410 may be a function f RGB may be applied to the feature coding of one of the RGB images 113. The CNN 400 may also include a branch having a normalization network 430 that may receive a positive image 440 and a negative image 450 (e.g., one of the 3D-CAD models stored in the database 128) as input data.

[0030] The output data of the RGB network 410 can be provided to RGB coded features 460, and the output data of the normalization network 430 can be provided to positive coded features 470 and negative coded features 480. A TripletLoss algorithm 490 (discussed with reference to Equation 1 above) can then be utilized to reduce the vector distance between the anchor image 420 and the positive image 440, and to increase the vector distance between the anchor image 420 and the negative image 450.

[0031] It is contemplated that the normalization network 430 may be implemented as a "Siamese Network" that may work in concert to compute different vector distances for the formed positive image 440 and negative image 450. It is also contemplated that the RGB network 410 and the normalization network 430 may include different layers (i.e., convolutional layers, max pooling layers, smoothing layers, dropout layers, recognition layers, and dense layers) that utilize different training data depending on a given application.

[0032] 5 illustrates a CNN 500 that may be utilized during a testing phase or during run-time operation. The CNN 500 may also be designed using a "Siamese network," although it is contemplated that other network designs may be utilized depending on a given application. It is contemplated that the RGB network 510 and the normalization network 530 may include different variations of convolutional layers, max pooling layers, smoothing layers, dropout layers, and dense layers depending on a given application.

[0033] RGB network 510 (i.e., f RGB ) may be provided as input with an RGB image 520, which again may be an RGB image 113 received by the client system 110. The CNN 500 includes a normalization network 530 (f normalization), the RGB network 510 can provide output vector data to RGB coding features 550, and the normalization network 530 can provide output vector data to one or more normal coding features 560, 570.

[0034] The CNN 500 can then compute a set of distance vectors 580 from the feature vectors of the normalized image 540 and the feature vectors encoded in the RGB image 520 using the output vector data provided by the RGB coded features 550 and one or more normal coded features 560, 570. The CNN 500 can then select the distance vector with the shortest distance in the semantic space.

[0035] Again, it is contemplated that the CNN may be operable to detect an "invalid" condition, indicating that a predetermined step or sequence performed by the user may not match any step in the procedure. The CNN may determine that the invalid condition may have been the result of an incorrect sequence of operations performed by the user. By analyzing the RGB image 113 acquired by the client system 110, it is contemplated that the CNN may analyze critical portions of the procedure to recognize the invalid condition.

[0036] 6 illustrates an RGB image 600 that may be acquired by the client system 110 and sent to the server system 120. The controller 124 may process the RGB image 600 to detect unexpected portions due to invalid operations by the user. For example, the controller may apply an image cropping algorithm using the RGB image 600 for the presence or absence of each portion to generate a first bounding box 610, a second bounding box 620, a third bounding box 630, and a fourth bounding box 640. The CNN may then determine whether an invalid condition exists based on the presence or absence of each portion in the RGB image 600. Alternatively, the CNN may determine whether the user performed an incorrect step based on the presence or absence of each portion in the RGB image 600.

[0037] It is contemplated that the presence or absence of the portion in the RGB image 600 can be processed by a two-branch CNN network trained using the TripletLoss algorithm discussed in connection with FIG. 4. It is contemplated that the controller 124 can be operable to apply an image cropping algorithm using one or more 3D-CAD models stored in the database 128 to generate one or more 3D-bounding boxes (e.g., the first bounding box 610) for preprogrammable essential or critical portions. It is contemplated that the 3D-bounding boxes can be projected into image space based on the current pose when training and testing the CNN. It is also contemplated that the two-dimensional projection of the 3D-bounding boxes can be used by the CNN to define a region of interest in the RGB image 600 that may contain the particular portion. The portion can be generated by the CNN cropping the RGB image 600 based on the predefined region of interest and rendering the cropped image.

[0038] It is also contemplated that the CNN may use one of the multiple 3D bounding boxes (e.g., the first bounding box 610) as an anchor image. It is also contemplated that the CNN 400 may be trained to use a null stage recognition process using the anchor image and the negative and positive images provided by the database 128. During the training phase, the anchor image and the positive image may contain identical parts, whereas the negative image may contain parts that are not identical to the anchor image. Again, it is contemplated that the CNN 400 may be trained to use a TripletLoss algorithm using the anchor image, the positive image, and the negative image to determine when a part within a given region of interest is detected or not present.

[0039] Once trained, the CNN 500 can be utilized to provide detection of invalid or incorrect stages during a test phase or run-time operation. The test or run-time operation can crop a captured image (e.g., RGB image 113 received by client system 110) into various portions. In this case, the CNN can be operable to compare the portions in the image with corresponding areas in two or more normal map images. It is conceivable that one normal map image contains the portion and the other normal image does not. If the features extracted from the image are closer to the normal map image that contains the portion, the CNN can determine that the portion has been detected. If the features extracted from the captured image are closer to the normal map image that does not contain the portion, the CNN can determine that the portion has not been detected (i.e., "none").

[0040] Table 2 below shows the outputs that may be generated by the CNN 500 during the invalidation stage recognition process. [Table 2]

[0041] As shown, CNN 500 can utilize invalid stage recognition for two or more images (e.g., image 1, image 2, image 3, image 4). For example, Table 2 shows that CNN 500 can generate the following outputs for "image 1" from "part 1" to "part 5": "detected, detected, none, none, none". Based on these outputs, invalid stage recognition may be operable to map the outputs to standard repair stages. As shown, for image 1, image 2, and image 3, the invalid stage recognition process has determined that no invalid stages were performed. In other words, for image 1 to image 3, CNN 500 has determined that the user followed the correct sequence of repair stages.

[0042] However, for "Image 4", the invalid stage recognition process has determined that an "invalid state" has occurred. The invalid state output may have been generated because the user mistakenly performed an incorrect repair sequence. For example, the fourth stage in the repair sequence may request to "detect" "Part 1". The CNN 500 did not detect the presence of "Part 1", thus detecting an invalid sequence. Alternatively, an invalid state may be generated if the user mistakenly attempts to re-add a portion from the correct repair sequence. For example, "Part 1" may be detected as absent for Image 5 because "Part 1" is currently behind or occluded by an added portion (e.g., Part 5). The invalid stage recognition process may further output an "invalid state" because "Part 1" is not visible, as reconstruction of "Part 5" was not requested until a later repair sequence.

[0043] The processes, methods or algorithms disclosed herein can be transmitted to and / or implemented in a processor, controller or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the processes, methods or algorithms can be stored as data, logic and instructions executable by a controller or computer in many forms, including but not limited to information persistently stored on non-writable storage media, e.g., ROM devices, and information mutably stored on writable media, e.g., floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. The processes, methods or algorithms can also be implemented in software executable objects. Alternatively, the processes, methods or algorithms can be embodied in whole or in part using suitable hardware components, e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware components, software components and firmware components.

[0044] Although exemplary embodiments have been described above, it is not intended that these embodiments describe all possible forms of the invention. Rather, the terms used herein are for purposes of description rather than limitation, and it will be understood that various changes may be made without departing from the spirit and scope of the invention. Furthermore, features of various embodiments may be combined to form further embodiments of the invention.

Claims

1. 1. A method of convolutional neural network (CNN), comprising: generating anchor vectors in a semantic space in response to anchor images provided to a first CNN, the anchor images being two-dimensional RGB images, the first CNN including one or more first convolutional layers, one or more first max pooling layers, a first flattening layer, a first dropout layer, and a first fully connected layer; generating positive and negative vectors in the semantic space in response to negative and positive images provided to a second CNN, the negative images being a first 3D CAD image and the positive images being a second 3D CAD image, the second CNN including one or more second convolutional layers, one or more second max pooling layers, a second flattening layer, a second dropout layer and a second fully connected layer; applying a cross-domain deep metric learning algorithm operable to extract image features in the semantic space using the anchor vectors, the positive vectors, and the negative vectors; performing staging recognition by analyzing the extracted image features within the semantic space; The method includes:

2. 1. A method of convolutional neural network (CNN), comprising: generating anchor vectors in a semantic space in response to anchor images provided to a first CNN, the anchor images being two-dimensional RGB images, the first CNN including one or more first convolutional layers, one or more first max pooling layers, a first flattening layer, a first dropout layer, and a first fully connected layer; generating positive and negative vectors in the semantic space in response to negative and positive images provided to a second CNN, the negative images being a first 3D CAD image and the positive images being a second 3D CAD image, the second CNN including one or more second convolutional layers, one or more second max pooling layers, a second flattening layer, a second dropout layer and a second fully connected layer; applying a cross-domain deep metric learning algorithm operable to extract image features in the semantic space using the anchor vectors, the positive vectors, and the negative vectors; determining whether an invalid repair sequence occurs based on an analysis of the image features within the semantic space; The method includes:

3. 3. The method of claim 1 or 2, wherein the cross-domain deep distance learning algorithm is a TripletLoss algorithm operable to decrease a first distance between the anchor vector and the positive vector in the semantic space and to increase a second distance between the anchor vector and the negative vector in the semantic space.

4. 3. The method of claim 1, wherein the one or more first convolutional layers and the one or more second convolutional layers are operable to apply one or more activation functions.

5. The method of claim 4 , wherein the one or more activation functions are implemented using rectified linear units.

6. The method of claim 4 , wherein the first CNN and the second CNN further include one or more normalization layers.

7. The method of claim 1 or 2, wherein the second CNN is designed using a Siamese network.

8. The method of claim 1 or 2, wherein the first CNN and the second CNN utilize a skip connection architecture.

9. 1. An augmented reality system, comprising: a visualization device operable to acquire one or more RGB images; A controller; Including, The controller is operable to generate anchor vectors in a semantic space in response to anchor images provided to a first CNN, the anchor images being two-dimensional RGB images, the first CNN including one or more first convolutional layers, one or more first max pooling layers, a first flattening layer, a first dropout layer, and a first fully connected layer; The controller is operable to generate positive and negative vectors in the semantic space in response to negative and positive images provided to a second CNN, the negative images being a first 3D CAD image and the positive images being a second 3D CAD image, the second CNN including one or more second convolutional layers, one or more second max pooling layers, a second flattening layer, a second dropout layer and a second fully connected layer; the controller is operable to apply a cross-domain deep metric learning algorithm operable to extract image features in the semantic space using the anchor vectors, the positive vectors, and the negative vectors; The controller is operable to determine a current stage of a work procedure. Augmented reality system.

10. The augmented reality system of claim 9 , wherein the controller is further operable to determine a pose of an image object in the one or more RGB images.

11. 10. The augmented reality system of claim 9, wherein the controller is further operable to decrease a first distance between the anchor vector and the positive vector in the semantic space and to increase a second distance between the anchor vector and the negative vector in the semantic space.

12. The augmented reality system of claim 9 , wherein the controller is further operable to display instructions on the visualization device based on the current stage of the work procedure.

13. The augmented reality system of claim 9 , wherein the second CNN is designed using a Siamese network.

14. 1. A method of augmented reality, comprising: generating anchor vectors in a semantic space in response to anchor images provided to a first CNN, the anchor images being two-dimensional RGB images, the first CNN including one or more first convolutional layers, one or more first max pooling layers, a first flattening layer, a first dropout layer, and a first fully connected layer; generating positive and negative vectors in the semantic space in response to negative and positive images provided to a second CNN, the negative images being a first 3D CAD image and the positive images being a second 3D CAD image, the second CNN including one or more second convolutional layers, one or more second max pooling layers, a second flattening layer, a second dropout layer and a second fully connected layer; extracting one or more image features from different modalities using the anchor vectors, the positive vectors, and the negative vectors; performing staging recognition by analyzing the extracted image features within the semantic space; The method includes:

15. 1. A method of augmented reality, comprising: generating anchor vectors in a semantic space in response to anchor images provided to a first CNN, the anchor images being two-dimensional RGB images, the first CNN including one or more first convolutional layers, one or more first max pooling layers, a first flattening layer, a first dropout layer, and a first fully connected layer; generating positive and negative vectors in the semantic space in response to negative and positive images provided to a second CNN, the negative images being a first 3D CAD image and the positive images being a second 3D CAD image, the second CNN including one or more second convolutional layers, one or more second max pooling layers, a second flattening layer, a second dropout layer and a second fully connected layer; extracting one or more image features from different modalities using the anchor vectors, the positive vectors, and the negative vectors; determining whether an invalid repair sequence occurs based on an analysis of the image features within the semantic space; The method includes:

16. 16. The method of claim 14 or 15, further comprising applying a TripletLoss algorithm operable to decrease a first distance between the anchor vector and the positive vector in the semantic space and to increase a second distance between the anchor vector and the negative vector in the semantic space.

Images