Self-authenticating augmented reality detection of components

Abstract

A system and method for self-verifying augmented reality detection of parts are provided, comprising receiving an image of a part in a real-world environment; determining a virtual camera pose in a CAD environment relative to a CAD model of the part, the virtual camera pose being substantially the same as a camera pose relative to the part in world space; generating a CAD image with the virtual camera pose; removing a background from the CAD image; generating a synthetic image by overlaying the masked CAD image onto the image of the part in the real-world environment; and feeding the synthetic image to a trained locking model to determine whether the CAD image is locked to a captured image of the part in the real-world environment, wherein the locking model is trained on a dataset including ground truth synthetic image examples of locked and unlocked parts.

Description

Technical Field

[0001] This disclosure relates in general to aircraft inspection, and more specifically to the inspection of manufactured or assembled parts of aircraft using augmented reality and artificial intelligence. Background Technology

[0002] In aircraft manufacturing, inspections of manufactured or assembled parts are performed to identify defects and anomalies, and such inspections can be aided by tools such as tablet computers or other devices. However, existing inspection aids for visually inspecting aircraft components and their installation on the aircraft or its sub-components require human intervention, such as ensuring that one or more components being inspected are correctly oriented and positioned to verify the inspection results. While human expert inspection is a reliable method, it is still limited by human perception, attention, and judgment, and can be extremely labor-intensive and time-consuming. The ability to promptly identify and verify the inspection results of aircraft components and component installations can improve the efficiency and accuracy of inspections, and minimize potential delays due to rework. Furthermore, additional automated inspection aids are needed, which can be used at the manufacturer's factory and other remote work sites where expertise regarding potential problems and inspection standards may be less than at the manufacturer's factory. Summary of the Invention

[0003] In view of the above, according to one aspect, a system for self-verifying augmented reality detection of a part is provided, comprising a processing circuitry system communicatively coupled to a non-volatile memory and instructions, which, when executed by the processing circuitry system, cause the processing circuitry system to: receive an image of a part in a real-world environment captured by a camera oriented in a camera pose relative to the part in world space; determine a virtual camera pose of a CAD model of the part in a CAD environment substantially the same as the camera pose relative to the part in world space; generate a CAD image via the virtual camera oriented in the virtual camera pose; generate a masked CAD image of the part by removing a background from the CAD image; generate a synthetic image by overlaying the masked CAD image onto an image of the part in the real-world environment; provide the generated synthetic image to a trained latching model to determine whether the CAD image is latched to a captured image of the part in the real-world environment, the trained latching model having been trained on a dataset including ground truth synthetic image examples of locked and unlocked parts; and output the latching determination.

[0004] The features, functions, and advantages already discussed can be implemented independently in various embodiments or combined in other embodiments, further details of which can be seen in the following description and figures. Attached Figure Description

[0005] Figure 1 This is an overall schematic diagram illustrating an overview of a system for self-verifying augmented reality detection of components according to an embodiment of the present disclosure, showing both training and inference using a locked model.

[0006] Figure 2 Detailed illustrations Figure 1 A schematic diagram illustrating the training of the system's locking model.

[0007] Figure 3 Further illustration Figure 1 A schematic diagram of the training of the locking model.

[0008] Figure 4 yes Figure 1 An illustration of the augmented reality device of the system, showing a lock confirmation indicator.

[0009] Figure 5 The diagram illustrates the process of obtaining... Figure 4 The affirmative lock in the data determines the time series of previously captured synthetic images.

[0010] Figure 6A and Figure 6B A flowchart of a method for training a locking model according to embodiments of the present disclosure is provided.

[0011] Figure 6C The diagram illustrates the use of Figure 6A and Figure 6B The method steps for performing inference using a trained locking model.

[0012] Figure 7 The examples applied according to this disclosure can be Figure 1 The system and Figures 6A to 6C The diagram shows the aircraft detected by the method.

[0013] Figure 8 The diagram is for implementation. Figure 1 A schematic diagram of an exemplary computing system. Detailed Implementation

[0014] In view of the above problems, such as Figure 1As shown, a computing system 10 is provided for self-verifying augmented reality detection of a component 24 mounted in an installation environment. Component 24 is illustrated in a simplified manner as a bracket or bracket attachment. However, component 24 can be any part, sub-assembly, or structure used in another product or application. In one example application, component 24 is an aircraft or spacecraft component. The installation environment in which the component is mounted is typically a larger superstructure or assembly to which the component is mounted, and therefore can be, for example, an aircraft component assembly or a spacecraft component assembly.

[0015] As shown, system 10 includes a processing circuitry system 12 communicatively coupled to non-volatile memory 14 and instructions 18 stored therein, which, when executed by the processing circuitry system 12, cause the processing circuitry system 12 to perform the functions described herein. The computing system 10 may take the form of a computing device 11 (which is mobile or handheld and includes a built-in camera 22), such as a smartphone, tablet computing device, head-mounted augmented or virtual reality device, or laptop computer. Alternatively, the computing device 11 may be a personal computer, server, or other computing device, and the camera 22 may be a peripheral or remote camera connected to the computing device via a data connection such as Bluetooth or Wi-Fi network connection. In either configuration, a user can navigate a real-world environment to capture images of components installed in the environment.

[0016] Figure 1 The diagram illustrates both the training and inference phases performed by the computing system 10, referred to as training time and inference time. First, a description of the inference time using the trained locking model 40A will be provided, followed by a description of the training time, during which the locking model 40 is trained to generate the trained locking model 40A.

[0017] During inference time, the processing circuitry 12 is configured to receive captured images 20A of components 24 in the real-world environment 20, captured by camera 22, which is oriented in a camera pose 26 relative to components 24 in world space, referring to the three-dimensional space within the real-world environment 20. Images of components 24 in the real-world environment 20 can be received via image acquisition module 34. As used herein, camera pose refers to orientation in a three-dimensional coordinate system (such as world space or virtual space) and can therefore be represented, for example, by a six-degree-of-freedom (6DOF) vector containing the x, y, z, pitch, roll, and yaw values ​​for each camera. The camera pose can be determined with reference to signals from the internal inertial measurement unit (IMU) 13 of the computing system 10, and alternatively or additionally, the position of camera 22 in the world space within the real-world environment 20 can be determined using simultaneous localization and mapping techniques based on environment-based camera images.

[0018] Continuing with the reasoning process, the processing circuitry 12 may include a CAD image processing module 36 for determining a virtual camera pose 30 of the virtual camera 28 relative to the CAD model 16 of the component in the CAD environment (virtual space), which is substantially the same as the camera pose 26 relative to the component 24 in world space within the real-world environment 20. A CAD image 16A of the CAD model is generated via the virtual camera 28 oriented with the virtual camera pose 30. The CAD image 16A is a two-dimensional projection of the three-dimensional CAD model 16 taken from the perspective of the virtual camera 28 with the virtual camera pose 30. The CAD image processing module 36 generates a masked CAD image 16A of the component 24 by removing the background from the CAD image 16A. The mask is a transparent mask such that only the CAD model 16 is visible in the masked CAD image 16A, and the background is removed.

[0019] The processing circuitry system 12 may include a composite image generator 38 that receives a masked CAD image 16A of component 24 from CAD image processing module 36, and also receives a captured image 20A of component 24 in the real-world environment 20 from camera 22 via image acquisition module 34. The composite image generator 38 then generates a composite image 50 by overlaying the masked CAD image 16A onto the captured image 20A of component 24 in the real-world environment 20.

[0020] Additionally, during inference time, the CAD image processing module 36 provides the generated composite image 50 to the trained locking model 40A, thereby making a locking determination 23 regarding whether the CAD image in the composite image is locked to the captured image of the component 24 in the real-world environment 20. The locking determination 23 is passed from the trained locking model 40A to the graphical user interface module (GUI) 41. The GUI module 41 also receives the composite image 50 from the composite image generator 38. The GUI module 41 generates a graphical indication 23A of the locking determination 23. The GUI module 41 outputs the graphical indication 23A of the locking determination 23 and the composite image 50 to the display 58. In the depicted example, the locking determination 23 is affirmative, and the graphical indication is a checkmark indicating a positive determination. An example of a negative locking determination is as follows.

[0021] In this way, the display 58 can show an augmented reality view of the captured image 20A of the real environment 20, with the CAD image 16A superimposed on the composite image 50, and further with a graphic indication 23A of lock confirmation 23. In use, the display 58 can be configured to update the composite image in real time or near real time, allowing a user holding the computing device 11 to adjust the position of the device and camera to align the CAD image 16A with the component 24 in the real world 20, enabling a positive lock confirmation.

[0022] In some configurations, a negative lock determination can be output and the defect indicated on the display of the computing device, such as when a negative lock determination indicates a missing or incorrectly installed part. For this reason, the computing system described herein is referred to as “self-verifying.” However, in another configuration of the system, a defect model 25 (shown in dashed lines) can be provided. In this configuration, when the lock determination is positive, as determined at decision box 39, a synthetic image 50 (or alternatively captured image 20A) can also be passed to a trained defect model 25, which has been trained to identify defects in part 24 in the synthetic image 50 (or alternatively captured image 20A). The trained defect model 25 has been trained using synthetic images 50 (or alternatively captured images 20A) of part 24 taken from a specific predetermined angle (i.e., from a predetermined defect model training time camera pose), each image having a ground truth defect label. In the configuration utilizing the defect model, the locking determination 23 described herein is technically important because it enables the defect model 25 to accurately classify its received images as defective or defect-free by ensuring that the synthetic image 50 (or the captured image 20A) is taken from a camera pose similar to or the same as the original image on which the defect model was trained. For this reason, the virtual camera pose 30 used to generate the training dataset 54 is chosen to be the same as or substantially similar to the camera pose during the defect model training time. In the configuration utilizing the defect model 25, the computational system is self-verifying because it verifies the captured or synthetic images sent to the defect model to increase the accuracy of the classification performed by the defect model 25.

[0023] The process circuitry 12 is now directed to training the trained locking model 40A. It is further configured to implement a training data module 48, which generates a training dataset 54 for training the locking model 40 during training time prior to inference time. During training time, an untrained (or, in the case of pre-training, incompletely trained) locking model 40 is trained on the training dataset 54, which includes ground truth data 56 comprising synthetic images 50 and associated ground truth labels 46 indicating whether each synthetic image 50 shows a real-world locking CAD image of part 24 and the captured image 20A (i.e., the training dataset 54 of synthetic images where each image is labeled “locked” or “unlocked”), thereby producing the trained locking model 40A. The untrained (or incompletely trained) locking model 40 includes an untrained locking classifier 42, which may include an untrained neural network 44. Therefore, the trained locking model 40A includes a trained locking classifier 42A that utilizes a trained neural network 44A that has been trained to output a locking determination 23, which can then be output to a display 58 in the form of a graphical indication 23A of the locking determination 23.

[0024] Figure 2 Illustrations are provided. Figure 1 A schematic diagram of the training of the locking model 40 of the computing system 10 is shown. As shown, the training data module 48 is configured to implement the training dataset 54 generation algorithm as follows. The training data module sends an image request to the image acquisition module 34. In response, the image acquisition module 34 acquires the captured image 20A and returns the camera pose 26 of the camera 22 for the captured image 20A. The camera pose 26 can be determined based on the metadata of the IMU 13 as described above, or the camera pose 26 can be inferred using image processing techniques on the captured image 20A. In another configuration, the camera pose 26 can be output by a robotic camera system that can programmatically position the camera to the desired pose.

[0025] In some detection schemes, a locking model 40 is trained using multiple arbitrary camera poses 26 to detect locks from any arbitrary camera pose 26 relative to the part 24 being detected during detection. This approach provides sufficient quality when detection only detects the presence or absence of a part in the environment with appropriate position and orientation. In other detection schemes requiring more precise alignment, such as when inspecting hard-to-detect defects (e.g., oxidation, paint chips, scratches, or dents), one or a set of pre-defined camera poses 26 can be predetermined, allowing training and inference to be performed using one or more predefined camera poses 26. This approach can improve the accuracy of the computational system.

[0026] To generate training dataset 54, training data module 48 is configured to receive training-time captured images of training-time components in a real-world environment. Training data module 48 passes the camera pose 26 of the training-time captured images to virtual pose generation logic 31, which in turn commands CAD image processing module 36 to generate multiple CAD images 16A. First, virtual pose generation logic 31 determines a matching virtual pose 30A within the CAD programming environment that is substantially identical to the camera pose 26 of the component in the training-time captured images in world space. Then, virtual pose generation logic 31 sends the matching virtual pose 30A along with commands to CAD image processing module to generate a matching CAD image 16A1, which includes the CAD model captured by the matching virtual camera pose 30A in the CAD programming environment. The matching CAD image 16A1 is generated first, and then a mask is created and the background is removed, resulting in a masked CAD image 16A1 with a transparent background. The masked CAD image 16A1 is overlaid on the captured image 20A in the manner described above to generate a locked composite image 50A1, which can be labeled with a positive ground truth tag 46A1. It should be understood that the CAD image 16A can be generated as completely opaque or partially transparent as needed. When opaque, they can be generated to be photorealistic by applying lighting, textures, etc., that match the real-world environment 20.

[0027] To generate negative training example training data pairs, for each pair, the image acquisition module receives training-time captured images of the training-time components in the real-world environment from the aforementioned positive training example data pairs. To provide robust negative examples for training, the training data module 48 is configured to generate various mismatched virtual poses 30B that do not match the camera pose 26. The training data module 48 is configured to send each of the various mismatched poses 30B, along with a command to generate a CAD image for each mismatched pose, to the CAD image processing module 36. In response, the CAD image processing module 36 generates negative examples, such as mismatched CAD images 16A2 and 16A3, and generates masked mismatched CAD images 16A2, 16A3 from each by removing the background. As shown, the composite image generator 38 is configured to overlay these masked mismatched CAD images 16A2, 16A3 onto corresponding instances of the same captured image 20A. Thus, for the same captured image 20A, at least one positive training example (labeled as locked synthetic image 50A1) and one or more negative training examples (labeled as unlocked synthetic images 50A2, 50A3) are generated. After generating these training examples, they can be presented to human experts for human input at the human-in-the-loop verification interface 71 to verify the ground truth label. The verified synthetic image 50 and label 46 are returned to the training data module 48 as training dataset 54. Alternatively, training dataset 54 can be exported to training data module 48 without human verification. In this case, a procedural verification process can be used. This process can be repeated for multiple captured images 20A taken from different camera poses 26 and / or under different environmental conditions (lighting, temperature, air quality, condensation, etc.). In this way, a verified training dataset 54 can be generated using positive and negative examples under various conditions.

[0028] Go to Figure 3 The illustration shows the use of... Figure 2 A schematic diagram of training the locking model 40 on the generated training dataset 54 is shown. The training dataset 54 includes training data pairs 82. Each training data pair 82 includes a synthetic image 50 and a corresponding ground truth classification label 46, such as a positive classification label (i.e., locked) or a negative classification label (i.e., unlocked). Each synthetic image 50A can be input to the neural network as a 2D array. In other embodiments, multiple channels can be used, such as a 2D (image) pixel array for more than one channel (such as three channels (e.g., RGB) or four channels (e.g., RGB-D)). However, for the sake of simplicity, this document describes an embodiment where the input images are 2D arrays.

[0029] Once the synthetic image 50A is input as a 2D array into the neural network 44, the 2D array is passed through one or more convolutional layers (for feature mapping) and pooling layers (for reducing the dimension of the downstream fully connected hidden layers by reducing the dimension of the feature maps generated via the convolutional layers). Before being passed to the fully connected hidden layers, the 2D array is flattened into a 1D array and then passed to the fully connected hidden layers and the output layer consisting of (locked and unlocked) classifications to produce predicted classification labels 46A (e.g., locked or unlocked). For each prediction based on each input, the predicted classification 46A is compared with the ground truth classification label 46 to determine if any error exists. Therefore, a loss function is applied and the loss is computed between the ground truth classification label 46 (ground truth) of each synthetic image 50 and the predicted classification label 46A of that synthetic image. The backpropagation algorithm can be used to train the neural network by adjusting the weights between nodes in each layer of the fully connected hidden layers of the neural network 44 to minimize the error. This process is repeated for each training data pair 82 in the training dataset 54. The result is a trained locking model 40A, including a trained neural network 44A used as a trained locking classifier 42A. Once trained as described, the trained locking model 40A includes a neural network 44 configured to receive the generated synthetic image at inference time and, in response, classify the generated synthetic image as locked or unlocked.

[0030] Now go to Figure 4 The illustration shows the imaging of the real-world environment 20. Figure 1 An example of computing device 11. Components 24A-24C are mounted in real-world environment 20. The computing device 11 in this embodiment can be configured as a mobile computing device, such as a smartphone or tablet computer, having a rear camera serving as camera 22 and a front display serving as display 58. The computing device 11 is shown capturing images of two different locations in real-world environment 20 and displaying a composite image 50 for each location. As described above, the composite image 50 includes a masked CAD image 16A and an image of component 24 in the real-world environment. The composite image 50 is provided to a trained locking model 40, whose output indicates whether the captured image of CAD image 16A is locked or unlocked to component 24 in real-world environment 20. Based on this locking determination, the computing device 11 is configured to display a graphical indication 23A of the locking determination on display 58.

[0031] Figure 4An example with a positive lock graphic icon 23A1 is illustrated at (A). To make a positive lock determination, the trained lock model 40A detects that the CAD image 16A substantially or completely overlaps with an image of part 24A in the real-world environment 20, and that the boundaries of each of the parts 24A and the projected CAD models in the CAD image 16A have similar dimensions and shapes. Once a synthetic image 50 with a positive lock determination is obtained, the synthetic image 50 can be stored in a storage device as evidence of the existence of the locked part, and / or transferred to… Figure 1 Defect model 25 is used for defect analysis.

[0032] Figure 4 The negative lock icon 23A2 is shown at (B). Figure 4 In the process, the locking model makes a negative locking determination by detecting that the boundaries between CAD image 16C and part 24C do not overlap or have similar dimensions and shapes. A navigation icon 96 is provided to instruct the user on the direction to move the computing device to align part 24C in the captured image with CAD image 16A. As shown, the upward-pointing navigation icon 96A is highlighted to instruct the user to move the computing device with the camera upwards in an attempt to align part 24C and CAD image 16C.

[0033] Figure 5 The diagram illustrates the process of obtaining... Figure 4 A definitive lock at (A) establishes the time series of the previously captured composite image 50. (As...) Figure 5 As shown in (A)-(C), in this example, a navigation icon is displayed on or near the CAD image 16A. This navigation icon includes an arrow pointing in the direction in which the user should move the camera to align the CAD image with part 24. Once a lock is made, as... Figure 4 As shown at (D), the graphic icon 23A1 is definitely locked.

[0034] Figures 6A-6C The illustration shows a method for training and performing inference using a locked model according to an example embodiment of the present disclosure. Figure 6A and Figure 6B The diagram illustrates the steps involved in training a locking model, while Figure 6C The diagram illustrates the steps of a method for performing inference using a trained locking model. It should be understood that this method can be used as described above and... Figures 1-5 The method is implemented using the software and hardware components shown. It should be understood that method 100 can also be implemented using other suitable hardware and software components in other contexts. In particular, Figure 6A and Figure 6B The training time method steps can be found in the same way as Figure 6CThe steps in the process are executed on different computing devices, and therefore the processing circuitry used to execute these steps can be distributed across different computing devices.

[0035] During training, method 100 includes generating a training dataset at least in part by generating positive example training data pairs in 102. Typically, step 102 is performed multiple times to generate multiple positive example training data pairs. Each positive example training data pair is generated at least in part by performing steps 104-114. In 104, method 100 includes receiving a training-time captured image of a part in a real-world environment. In 106, method 100 includes determining a matching virtual pose within a CAD program environment, the matching virtual pose being substantially identical to a camera pose in world space relative to the part in the training-time captured image. In 108, the method includes generating a matching CAD image comprising a projection of a CAD model captured by a virtual camera in the matching virtual camera pose within the CAD program environment. In 110, the method includes generating a masked CAD image from the matching CAD image. At 112, method 100 includes overlaying the masked CAD image onto the training-time captured image to generate a locked synthetic image. Furthermore, at step 114, method 100 includes tagging the locked synthetic image with a positive ground truth label. From step 114, the method proceeds to... Figure 6B 116 in the middle.

[0036] At 116, method 100 further includes generating a training dataset at training time, at least in part, by generating multiple negative example training data pairs. Typically, step 116 is performed at least once for each positive training data pair generated at 102 to generate a corresponding negative training example data pair. Each negative training data pair is generated by performing at least steps 118-128. At 118, the method includes receiving a training-time captured image of a training-time component in the real-world environment from the positive training example data pair. At 120, the method includes generating a mismatched virtual pose that does not match the camera pose of the training-time captured image. If desired, random sampling (or random generation) of the mismatched virtual camera pose may be performed each time step 120 is passed to generate various mismatched virtual poses at 120. At 122, the method includes generating a mismatched CAD image with the mismatched virtual pose. At 124, the method includes generating a masked mismatched CAD image from the mismatched CAD image. In 126, the method includes overlaying a masked, mismatched CAD image onto the captured image to generate an unlocked synthetic image. In 128, the method includes tagging the unlocked synthetic image with a negative locked ground truth label.

[0037] In 130, the method further includes training a locking model on a training dataset comprising both positive and negative example training data pairs using a backpropagation algorithm to generate a trained locking model. Alternatively, in an alternative training process, only positive example training data pairs can be used to train the locking model. From 130, the method proceeds to... Figure 6C As mentioned above, the locking model can include a convolutional neural network and can be configured as a binary classifier.

[0038] Figure 6C The diagram illustrates the method steps for self-verifying augmented reality detection of components, executed at inference time, and can be used in conjunction with... Figures 6A-6B The steps in the method are performed on different computing devices. At 132, method 100 includes receiving, at inference time, an image of a part in a real-world environment captured by a camera oriented in a camera pose relative to the part in world space. At 134, the method includes determining a virtual camera pose relative to a CAD model of the part in a CAD environment, the virtual camera pose being substantially the same as the camera pose relative to the part in world space. At 136, the method includes generating a CAD image via a virtual camera oriented in the virtual camera pose. At 138, the method includes generating a masked CAD image of the part by removing a background from the CAD image. At 140, the method includes generating a synthetic image by overlaying the masked CAD image onto an image of the part in the real-world environment. At 142, the method includes providing the generated synthetic image to a trained locking model to determine whether the CAD image is locked to a captured image of the part in the real-world environment. It should be understood that the trained locking model has been trained on a training dataset comprising multiple training dataset pairs in the manner described above, each training dataset pair including a synthetic image and ground truth labels indicating whether the synthetic image depicts a locked or unlocked part and a CAD image. At 144, the method also includes outputting a lock determination. As shown at 146, the output may include displaying an image of the part in the real-world environment along with a graphical indication of the lock determination. The displayed graphical indication may be a positive lock graphical indication or a negative lock graphical indication, examples of which are shown in... Figure 4 As shown, this depends on the lock determination.

[0039] As described above, the trained locking model includes a neural network configured to receive a generated synthetic image at inference time and, in response, classify the generated synthetic image as locked or unlocked. The neural network may include one or more convolutional layers and multiple fully connected hidden layers, as described above. Additionally, the neural network may be configured to include a binary classifier with an output layer coupled to the output side of the hidden layers, the output layer including at least one node corresponding to a confirmed lock determination, as described above.

[0040] Now let's turn to the application of the above technologies. Figure 7 This is an illustration of aircraft 300, which incorporates [equipment / technology / mechanism]. Figures 1-5 The system and Figures 6A-6C The methods described herein are used to inspect components. For example, aircraft 300 includes mechanical fasteners, electrical connectors, structural components, fuselage panels, etc., which can be the components described above that are being inspected. These are merely illustrative examples, and it should be understood that other types of vehicles and components can also be inspected using the systems and methods described herein.

[0041] The systems and processes described herein have the potential benefit of enabling self-verifying augmented reality (AR) detection of parts and / or part assembly using artificial intelligence (AI). The described systems and processes include providing a self-verifying AR system by using an AI model trained with a classifier that determines whether a 3D CAD image overlaid on a real-life environment image is locked (overlapping with the part) onto the real-life view to make visual detection valid. Additionally, the systems include an AI model trained with a classifier that determines whether a synthetic image has been obtained, which can be used alone or in combination with a defect model to determine whether a defect is likely present in the part and / or part assembly.

[0042] Figure 8 A non-limiting embodiment of a computing system 400 capable of implementing one or more of the methods and processes described above is illustrated schematically. The computing system 400 is shown in a simplified form. The computing system 400 can embody the above and... Figure 1 The computing system 10 is illustrated in the figure. Components of the computing system 400 may be included in one or more of the following: personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, as well as wearable computing devices, such as smartwatches and head-mounted augmented reality devices.

[0043] The computing system 400 includes a processing circuit system 402, volatile memory 404, and non-volatile storage device 406. The computing system 400 may optionally include a display subsystem 408, an input subsystem 410, a communication subsystem 412, and / or... Figure 8 Other components not shown.

[0044] Processing circuitry system 402 typically includes one or more logic processors, which are physical devices configured to execute instructions. For example, a logic processor may be configured to execute instructions that are part of one or more of the following: applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transition the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0045] A logic processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the processing circuit system 402 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Components of the processing circuit system may optionally be distributed across two or more separate devices that may be remotely located and / or configured for collaborative processing. For example, aspects of the computing system disclosed herein may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. It is understood that, in this case, these virtualized aspects operate on different physical logic processors on various different machines. These different physical logic processors on different machines will be understood to be collectively covered by the processing circuit system 402.

[0046] The non-volatile storage device 406 includes one or more physical devices configured to store instructions that can be executed by a processing circuitry system to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 406 can be changed, for example, to store different data.

[0047] Non-volatile storage device 406 may include removable and / or built-in physical devices. Non-volatile storage device 406 may include optical memory, semiconductor memory, and / or magnetic memory, or other high-capacity storage device technologies. Non-volatile storage device 406 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 406 is configured to retain instructions even when the non-volatile storage device 406 is powered off.

[0048] Volatile memory 404 may include a physical device containing random access memory. Volatile memory 404 is typically used by processing circuitry system 402 to temporarily store information during software instruction processing. It should be understood that volatile memory 404 typically does not continue storing instructions when power is off.

[0049] The processing circuitry 402, volatile memory 404, and non-volatile storage device 406 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).

[0050] The terms "module," "program," and "engine" can be used to describe an aspect of a computing system 400, typically implemented in software by a processor, to perform specific functions using portions of volatile memory. These functions involve special configuration of the processor to perform translational processing for those functions. Therefore, a module, program, or engine can be instantiated using portions of volatile memory 404 via a processing circuitry system 402 that executes instructions stored in non-volatile storage device 406. It should be understood that different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass single or multiple sets of executable files, data files, libraries, drivers, scripts, database records, etc.

[0051] When a display subsystem 408 is included, it can be used to present a visual representation of data stored by the non-volatile storage device 406. The visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored in the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 408 can also be transformed to visually represent changes in the underlying data. The display subsystem 408 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the processing circuitry system 402, the volatile memory 404, and / or the non-volatile storage device 406 in a shared housing, or such display devices may be peripheral display devices.

[0052] When the input subsystem 410 is included, the input subsystem 410 may include one or more user input devices, such as a keyboard, mouse, touch screen, camera or microphone, or interface with them.

[0053] When a communication subsystem 412 is included, the communication subsystem 412 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 412 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured to communicate via wired or wireless local area networks or wide area networks, broadband cellular networks, etc. In some embodiments, the communication subsystem may allow the computing system 400 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0054] In addition, this disclosure includes configurations in accordance with the following terms.

[0055] Clause 1. A computational system for self-verifying augmented reality detection of a component, the system comprising: a processing circuitry communicatively coupled to a non-volatile memory storing instructions, the instructions, when executed by the processing circuitry, causing the processing circuitry to: receive an image of the component in a real-world environment captured by a camera oriented in a camera pose relative to the component in world space; determine a virtual camera pose in a CAD environment relative to a CAD model of the component, the virtual camera pose being substantially the same as the camera pose relative to the component in the world space; generate a CAD image via a virtual camera oriented with the virtual camera pose; and obtain the image from the CAD model. The process involves: removing the background from an image to generate a masked CAD image of the component; generating a synthetic image by overlaying the masked CAD image onto the image of the component in the real-world environment; providing the generated synthetic image to a trained locking model to determine whether the CAD image is locked to a captured image of the component in the real-world environment, the trained locking model being trained on a training dataset comprising multiple pairs of training datasets, each pair including a synthetic image and ground truth labels indicating whether the synthetic image depicts a locked or unlocked component and CAD image; and outputting the locking determination.

[0056] Clause 2. The system according to Clause 1 further includes a display configured to display together the image of the component in the real-world environment and an indication of the lock determination associated with the component.

[0057] Clause 3. The system according to any one of Clauses 1 to 2, wherein outputting the lock determination includes outputting a graphical indication to a display whether the CAD image is locked or unlocked to the component in the captured image of the real-world environment.

[0058] Clause 4. The system according to any one of Clauses 1 to 3, wherein the trained locking model comprises a neural network configured to receive the generated synthetic image at inference time and, in response, classify the generated synthetic image as locked or unlocked.

[0059] Clause 5. The system according to Clause 4, wherein the neural network comprises one or more convolutional layers.

[0060] Clause 6. The system according to Clause 5, wherein the neural network further includes a plurality of fully connected hidden layers coupled to the convolutional layer on the input side.

[0061] Clause 7. The system according to Clause 6, wherein the neural network is configured to include a binary classifier comprising an output layer coupled to the output side of the hidden layer, the output layer comprising at least one node corresponding to a positive lock determination.

[0062] Clause 8. The system according to any one of Clauses 1 to 7, wherein the processing circuitry is further configured to: at training time, generate the training dataset at least in part by generating positive example training data pairs by: receiving training-time captured images of training-time components in a real-world environment; determining a matching virtual pose within a CAD program environment, the matching virtual pose being substantially identical to the camera pose of the component in world space relative to the training-time captured images; generating a matching CAD image comprising a projection of the CAD model captured by the virtual camera in the matching virtual camera pose within the CAD program environment; generating a masked CAD image from the matching CAD image; overlaying the masked CAD image onto the training-time captured images to generate a locked synthetic image; and labeling the locked synthetic image with a positive locked ground truth label.

[0063] Clause 9. The system according to Clause 8, wherein the processing circuitry is further configured to: during training time, generate the training dataset at least in part by generating a plurality of negative example training data pairs at least in part by: for each negative training data pair: receiving a training-time captured image of the training-time component in the real-world environment from the positive training example data pair; generating a mismatched virtual pose that does not match the camera pose of the training-time captured image; generating a mismatched CAD image with the mismatched virtual pose; generating a masked mismatched CAD image from the mismatched CAD image; overlaying the masked mismatched CAD image onto the captured image to generate an unlocked synthetic image; and labeling the unlocked synthetic image with a negative locked ground truth label.

[0064] Clause 10. The system according to Clause 9, wherein the processing circuitry is further configured to train the trained locking model on the training dataset comprising the positive example training data pairs and the negative example training data pairs using a backpropagation algorithm.

[0065] Clause 11. A method for training a locking model for self-verifying augmented reality detection of a part, the method comprising: generating a training dataset at least in part during training time by: generating positive example training data pairs by: receiving training-time captured images of the part in a real-world environment; determining a matching virtual pose within a CAD program environment, the matching virtual pose being substantially identical to a camera pose of the part in world space relative to the training-time captured images; generating a matching CAD image comprising a projection of the CAD model captured by the virtual camera in the matching virtual camera pose within the CAD program environment; generating a masked CAD image from the matching CAD image; overlaying the masked CAD image onto the training-time captured images to generate a locked synthetic image; and labeling the locked synthetic image with a positive locking ground truth label.

[0066] Clause 12. The method according to Clause 11, further comprising: during training time, generating the training dataset at least in part by generating a plurality of negative example training data pairs at least in part by: for each negative training data pair: receiving a training-time captured image of the training-time component in the real-world environment from the positive training example data pair; generating a mismatched virtual pose that does not match the camera pose of the training-time captured image; generating a mismatched CAD image with the mismatched virtual pose; generating a masked mismatched CAD image from the mismatched CAD image; overlaying the masked mismatched CAD image onto the captured image to generate an unlocked synthetic image; and labeling the unlocked synthetic image with a negative locked ground truth label.

[0067] Clause 13. The method of Clause 12 further comprises: training the locking model on the training dataset including the positive example training data pairs and the negative example training data pairs using a backpropagation algorithm to generate a trained locking model.

[0068] Clause 14. The method according to any one of Clauses 11 to 13, wherein the locking model comprises a convolutional neural network.

[0069] Clause 15. A method for self-verifying augmented reality detection of a component, the method comprising: receiving an image of the component in a real-world environment captured by a camera, the camera being oriented in a camera pose relative to the component in world space; determining a virtual camera pose of a CAD model of the component in a CAD environment, the virtual camera pose being substantially the same as the camera pose relative to the component in the world space; generating a CAD image via a virtual camera oriented in the virtual camera pose; generating a masked CAD image of the component by removing a background from the CAD image; generating a synthetic image by overlaying the masked CAD image onto the image of the component in the real-world environment; providing the generated synthetic image to a trained locking model to make a locking determination of whether the CAD image is locked to a captured image of the component in the real-world environment, the trained locking model having been trained on a training dataset comprising multiple pairs of training datasets, each pair including a synthetic image and ground truth labels, the ground truth labels indicating whether the synthetic image depicts a locked or unlocked component and CAD image; and outputting the locking determination.

[0070] Clause 16. The method according to Clause 15 further includes: displaying the image of the component in the real-world environment together with the indication of the lock determination associated with the component.

[0071] Clause 17. The method according to any one of Clauses 15 to 16, wherein outputting the locking determination includes outputting a graphical indication to a display whether the CAD image is locked or unlocked to the component in the captured image of the real-world environment.

[0072] Clause 18. The method according to any one of Clauses 15 to 17, wherein the trained locking model comprises a neural network configured to receive the generated synthetic image at inference time and, in response, classify the generated synthetic image as locked or unlocked.

[0073] Clause 19. The method according to Clause 18, wherein the neural network comprises one or more convolutional layers and a plurality of fully connected hidden layers.

[0074] Clause 20. The method according to Clause 19, wherein the neural network is configured to include a binary classifier comprising an output layer coupled to the output side of the hidden layer, the output layer comprising at least one node corresponding to a positive lock determination.

[0075] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as various variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in a different order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0076] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of various processes, systems and configurations, as well as other features, functions, actions and / or properties disclosed herein, and any and all equivalents thereof.

Claims

1. A computational system for self-verifying augmented reality detection of components, the system comprising: A processing circuitry system communicatively coupled to a non-volatile memory storing instructions, which, when executed by the processing circuitry system, cause the processing circuitry system to: Receive images of the component in a real-world environment captured by a camera, the camera being oriented in a camera pose relative to the component in world space; Determine the virtual camera pose in the CAD environment relative to the CAD model of the component, wherein the virtual camera pose is substantially the same as the camera pose in the world space relative to the component; CAD images are generated via a virtual camera oriented with the virtual camera pose. A masked CAD image of the component is generated by removing the background from the CAD image; A composite image is generated by overlaying the masked CAD image onto the image of the component in the real-world environment; The generated synthetic image is provided to a trained locking model to determine whether the CAD image is locked to a captured image of the component in the real-world environment. The trained locking model has been trained on a training dataset comprising multiple pairs of training datasets, each pair including a synthetic image and ground truth labels, the ground truth labels indicating whether the synthetic image depicts a locked or unlocked component and CAD image. as well as The lock is confirmed.

2. The system of claim 1, further comprising a display configured to display together the image of the component in the real-world environment and an indication of the lock determination associated with the component.

3. The system of claim 1, wherein outputting the lock determination includes outputting a graphical indication to a display whether the CAD image is locked or unlocked to the component in the captured image of the real-world environment.

4. The system of claim 1, wherein the trained locking model comprises a neural network configured to receive the generated synthetic image at inference time and, in response, classify the generated synthetic image as locked or unlocked.

5. The system of claim 4, wherein the neural network comprises one or more convolutional layers.

6. The system of claim 5, wherein the neural network further comprises a plurality of fully connected hidden layers coupled to the convolutional layer on the input side.

7. The system of claim 6, wherein the neural network is configured as a binary classifier comprising an output layer coupled to the output side of the hidden layer, the output layer comprising at least one node corresponding to a positive lock determination.

8. The system of claim 1, wherein the processing circuitry is further configured to: During training, the training dataset is generated, at least in part, by the following: Generate positive example training data pairs using the following method: Receive images captured during the training time of the training time component in a real-world environment; Determine a matching virtual pose within the CAD program environment, the matching virtual pose being substantially the same as the camera pose of the component in an image captured in world space relative to the training time; Generate a matching CAD image, the matching CAD image comprising a projection of the CAD model captured by the virtual camera in the matching virtual camera pose within the CAD program environment; Generate a masked CAD image from the matched CAD image; The masked CAD image is overlaid on the image captured during training to generate a locked synthetic image; as well as The locked composite image is marked with a ground truth label that confirms the ground truth.

9. The system of claim 8, wherein the processing circuitry is further configured to: During training, the training dataset is generated, at least in part, through the following further steps: Multiple negative example training data pairs are generated, at least in part, through the following: For each negative training data pair: Receive the training-time captured image of the training-time component in the real-world environment from the positive training example data pair; Generate a mismatched virtual pose that does not match the camera pose of the image captured during the training time; Generate a mismatched CAD image using the mismatched virtual pose; Generate a masked version of the mismatched CAD image from the mismatched CAD image; The masked mismatched CAD image is overlaid onto the captured image to generate an unlocked composite image; as well as The unlocked composite image is marked with a negative ground truth label.

10. The system of claim 9, wherein the processing circuitry is further configured to train the trained locking model on the training dataset comprising the positive example training data pairs and the negative example training data pairs using a backpropagation algorithm.

11. A method for training a locking model for self-verifying augmented reality detection of parts, the method comprising: During training, the training dataset is generated, at least in part, through the following methods: Generate positive example training data pairs using the following method: Receive images captured during the training time of the training time component in a real-world environment; Determine a matching virtual pose within the CAD program environment, the matching virtual pose being substantially the same as the camera pose of the component in an image captured in world space relative to the training time; Generate a matching CAD image, the matching CAD image comprising a projection of the CAD model captured by the virtual camera in the matching virtual camera pose within the CAD program environment; Generate a masked CAD image from the matched CAD image; The masked CAD image is overlaid on the image captured during training to generate a locked synthetic image; as well as The locked composite image is marked with a ground truth label that confirms the ground truth.

12. The method of claim 11, further comprising: During training, the training dataset is generated, at least in part, through the following further steps: Multiple negative example training data pairs are generated, at least in part, through the following: For each negative training data pair: Receive the training-time captured image of the training-time component in the real-world environment from the positive training example data pair; Generate a mismatched virtual pose that does not match the camera pose of the image captured during the training time; Generate a mismatched CAD image using the mismatched virtual pose; Generate a masked version of the mismatched CAD image from the mismatched CAD image; The masked mismatched CAD image is overlaid onto the captured image to generate an unlocked composite image; as well as The unlocked composite image is marked with a negative ground truth label.

13. The method of claim 12, further comprising: The locking model is trained on the training dataset, which includes the positive example training data pairs and the negative example training data pairs, using the backpropagation algorithm to generate the trained locking model.

14. The method of claim 11, wherein the locking model comprises a convolutional neural network.

15. A method for self-verifying augmented reality detection of components, the method comprising: Receive images of the component in a real-world environment captured by a camera, the camera being oriented in a camera pose relative to the component in world space; Determine the virtual camera pose in the CAD environment relative to the CAD model of the component, wherein the virtual camera pose is substantially the same as the camera pose in the world space relative to the component; CAD images are generated via a virtual camera oriented with the virtual camera pose. A masked CAD image of the component is generated by removing the background from the CAD image; A composite image is generated by overlaying the masked CAD image onto the image of the component in the real-world environment; The generated synthetic image is provided to a trained locking model to determine whether the CAD image is locked to a captured image of the component in the real-world environment. The trained locking model has been trained on a training dataset comprising multiple pairs of training datasets, each pair including a synthetic image and ground truth labels, the ground truth labels indicating whether the synthetic image depicts a locked or unlocked component and CAD image. as well as The lock is confirmed.

16. The method of claim 15, further comprising: The image of the component in the real-world environment is displayed together with the indication of the lock determination associated with the component.

17. The method of claim 15, wherein outputting the lock determination includes outputting a graphical indication to a display whether the CAD image is locked or unlocked to the component in the captured image of the real-world environment.

18. The method of claim 15, wherein the trained locking model comprises a neural network configured to receive the generated synthetic image at inference time and, in response, classify the generated synthetic image as locked or unlocked.

19. The method of claim 18, wherein the neural network comprises one or more convolutional layers and a plurality of fully connected hidden layers.

20. The method of claim 19, wherein the neural network is configured as a binary classifier comprising an output layer coupled to the output side of the hidden layer, the output layer comprising at least one node corresponding to a positive lock determination.

Images