Fine-grained visual recognition in mobile augmented reality

Abstract

The method includes: obtaining access to a three-dimensional point cloud representation of the object, including the pose of a scanning digital camera and corresponding video frames; downsampling the three-dimensional point cloud representation to obtain a set of candidate regions of interest; filtering the candidate regions of interest to select, as selected regions of interest, those having appearance changes that characterize different visual states based at least in part on the pose of the camera; generating region-of-interest images for the selected regions of interest from corresponding ones of the video frames; and training a deep learning recognition model based on the region-of-interest images. The trained deep learning recognition model can be used, for example, to determine the visual state of the object to direct repairs.

Description

[Technical field]

[0001] The present invention relates to electrical, electronic and computer technologies, and more particularly to augmented reality (AR) systems and the like. [Background technology]

[0002] Augmented reality (AR) enhances the perception of one's surroundings by overlaying media and graphics on top of what is seen in the real world. Over the last decade, major advances have been made and interest in AR has grown. Despite this progress, many AR users' experience remains primitive and lacks intelligence and automation, which makes user interaction rather unintuitive. AR allows tracking of virtual objects and annotations in physical space through computer vision techniques, but is not inherently intelligent to actually recognize the semantics of what is seen. For example, in the tech support domain, AR can recognize a laptop in point cloud form and enable tracked annotations on the laptop, but it does not actually know that a particular point cloud looks like a laptop unless the user labels it as such. AR would not know if the laptop's cover is open or closed, nor would it know that the laptop has had its heat sink removed, etc. Current technology requires that all interactions be driven by the user identifying what the user is looking at (e.g., pre-labeling the object, identifying the object's state, etc.) before the user can have the relevant AR content projected into the user's field of view, which severely limits the interaction. Thus, there was a need in the art to address the above-mentioned problems. Summary of the Invention

[0003] The method includes obtaining access to a three-dimensional point cloud representation of the object including a pose of a scanning digital camera and corresponding video frames; downsampling the three-dimensional point cloud representation to obtain a set of candidate regions of interest; filtering the candidate regions of interest to select those of the regions of interest having appearance changes characterizing different visual states as selected regions of interest based at least in part on the pose of the camera; generating region of interest images for the selected regions of interest from corresponding ones of the video frames; and training a deep learning recognition model based on the region of interest images.

[0004] Viewed from a further aspect, the invention provides a non-transient computer-readable medium comprising computer-executable instructions that, when executed by a computer, cause the computer to perform a method including: obtaining access to a three-dimensional point cloud representation of an object including a pose of a scanning digital camera and corresponding video frames; downsampling the three-dimensional point cloud representation to obtain a set of candidate regions of interest; filtering the candidate regions of interest to select those of the regions of interest having appearance changes characterizing different visual states as selected regions of interest based at least in part on the pose of the camera; generating region of interest images for the selected regions of interest from corresponding ones of the video frames; and training a deep learning recognition model based on the region of interest images.

[0005] Viewed from a further aspect, the invention provides an apparatus comprising a memory; and at least one processor coupled to the memory, operative to: obtain access to a three-dimensional point cloud representation of an object including a pose of a scanning digital camera and corresponding video frames; downsample the three-dimensional point cloud representation to obtain a set of candidate regions of interest; filter the candidate regions of interest to select those of the regions of interest having appearance changes characterizing different visual states as selected regions of interest based at least in part on the pose of the camera; generate region of interest images for the selected regions of interest from corresponding ones of the video frames; and train a deep learning recognition model based on the region of interest images.

[0006] Viewed from a further aspect, the present invention provides a computer program product for managing augmented reality, the computer program product comprising a computer readable recording medium readable by a processing circuit and storing instructions for the processing circuit to perform the method of the present invention.

[0007] Viewed generally from a further aspect, the present invention provides a computer program comprising software code portions stored on a computer readable medium and loadable into an internal memory of a digital computer when the program is run on the computer to perform the method of the present invention.

[0008] The present principles provide techniques for fine-grained visual recognition in mobile augmented reality. In one aspect, an exemplary method includes obtaining access to a three-dimensional point cloud representation of an object including a pose of a scanning digital camera and corresponding video frames; downsampling the three-dimensional point cloud representation to obtain a set of candidate regions of interest; filtering the candidate regions of interest to select those of the regions of interest having appearance changes that characterize different visual states as selected regions of interest based at least in part on the pose of the camera; generating region of interest images for the selected regions of interest from corresponding ones of the video frames; and training a deep learning recognition model based on the region of interest images.

[0009] As used herein, the act of "facilitating" includes performing an action, facilitating an action, assisting in the performance of an action, or causing an action to be performed. Thus, by way of example, and not by way of limitation, instructions executing on one processor may facilitate an action by sending appropriate data or commands that cause or assist in the execution of instructions on a remote processor. For the avoidance of doubt, when a human operator facilitates an action by doing something other than performing the action, the action is nevertheless performed by some entity or combination of entities.

[0010] One or more embodiments of the present invention, or elements thereof, may be implemented in the form of a computer program product including a computer-readable recording medium having computer usable program code for performing the indicated method steps. Furthermore, one or more embodiments of the present invention, or elements thereof, may be implemented in the form of a system (or apparatus) including a memory and at least one processor coupled to the memory and operable to perform the exemplary method steps. Still further, in another aspect, one or more embodiments of the present invention, or elements thereof, may be implemented in the form of a means for performing one or more of the method steps described herein; the means may include (i) a hardware module(s), (ii) a software module(s) stored in a computer-readable recording medium (or a number of such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii); any of (i)-(iii) implementing the specific techniques described herein.

[0011] The techniques of the present invention can provide substantial beneficial technical effects. For example, one or more embodiments may provide one or more of: fine-grained level visual recognition of the status of an item, such as a machine under repair, robustness to occlusion from a user's hand or tool, robustness to camera motion, and automatic ignoring of blurry images.

[0012] These and other features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

[0013] The present invention will now be described, by way of example only, with reference to a preferred embodiment as illustrated in the following drawings. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 illustrates a cloud computing environment in accordance with an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates abstract model layers according to an embodiment of the present invention. [Figure 3A] FIG. 3A illustrates an exemplary visual state of a computer that can be detected using an embodiment of the present invention. [Figure 3B] FIG. 3B illustrates an exemplary visual state of a computer that can be detected using an embodiment of the present invention. [Figure 3C] FIG. 3C illustrates an exemplary visual state of a computer that can be detected using an embodiment of the present invention. [Figure 3D] FIG. 3D illustrates an exemplary visual state of a computer that can be detected using an embodiment of the present invention. [Figure 4] FIG. 4 illustrates a combined data flow / block diagram according to an embodiment of the present invention. [Diagram 5] FIG. 5 illustrates an alternative combined data flow / block diagram according to an embodiment of the present invention. [Figure 6A] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 6B] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 6C] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 6D] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 6E] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 6F] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 6G] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 6H] FIG. 6 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 7] FIG. 7 illustrates an exemplary process for region of interest (ROI) according to an embodiment of the present invention. [Figure 8] FIG. 8 is a combined data flow / block diagram including run-time details according to an embodiment of the present invention. [Figure 9] FIG. 9 illustrates an exemplary model design according to the present invention. [Figure 10] FIG. 10 is a flow chart illustrating pose independent coarse grained recognition for selecting correct object models for repositioning according to an embodiment of the present invention. [Figure 11] FIG. 11 shows an example of ROI occlusion that can be effectively handled by embodiments of the present invention. [Figure 12] FIG. 12 illustrates how ROI optimization evaluates device movement speed, viewing angle, and distance to select an optimally captured ROI image according to an embodiment of the present invention. [Figure 13] FIG. 13 illustrates a confusion matrix of a state recognition result according to an aspect of the present invention. [Figure 14] FIG. 14 shows a comparison of recognition accuracy for data augmentation (DA) with different numbers of training images in accordance with an aspect of the present invention. [Figure 15] FIG. 15 shows the recognition accuracy with different ROI sizes according to the present invention. [Figure 16] FIG. 16 shows error rates and latencies for different numbers of trials in accordance with an aspect of the present invention. [Figure 17] FIG. 17 illustrates the accuracy of different models using either ROI capture or raw image frames in accordance with aspects of the present invention. [Figure 18] FIG. 18 shows a comparison of accuracy of different models as the object becomes more complex and more ROIs are used in accordance with an aspect of the present invention. [Figure 19] FIG. 19 shows the total number of parameters for different models with increasing ROI in accordance with an aspect of the present invention. [Figure 20] FIG. 20 shows exemplary accuracy of the lightweight (LW) convolutional neural network and discrete CNN-LSTM (DCL; LSTM = long short-term memory) models for five objects achieved in an exemplary embodiment of the present invention. [Figure 21] FIG. 21 is a table illustrating resource consumption under different modes achieved in exemplary embodiments of the present invention. [Figure 22] FIG. 22 illustrates a computer system that may be used to implement one or more aspects and / or elements of the present invention and that represents a cloud computing node. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Although the present invention includes detailed descriptions of cloud computing up front, it is understood that implementation of the teachings recited herein is not limited to a cloud computing environment, but rather the inventive environment can be implemented in combination with any other type of computing environment now known or later developed.

[0016] Cloud computing is a service delivery model for on-demand network access with the convenience of accessing a shared pool of rapidly provisioned and open configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) with minimal administrative effort or interaction with a service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0017] Its features are as follows: On-Demand Self-Service: Cloud consumers are automatically provisioned with computing power - server time and network storage - on an as-needed basis without the need for human interaction with the service provider. Pervasive Network Access: Capabilities are available over the network and accessed through standard mechanisms facilitating use by different thin- and thick-client platforms (eg, mobile phones, laptops, and PDAs). Resource Sharing: Using a multi-tenant model, a provider's computing resources are shared to serve multiple consumers, with different physical and virtualized resources dynamically allocated and reallocated as needed. A sense of location independence exists, such that consumers generally have no control or knowledge of the exact location (e.g., country, state, or data center) of the resources provided, but can specify the location at a higher layer of abstraction. Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, to quickly scale out and quickly release to quickly scale in. To the consumer, the capabilities available for provisioning often appear unlimited and can be purchased in any quantity at any time. Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at several levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of the services being used.

[0018] The service model is as follows: Software as a Service (SaaS): The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application functionality, except for limited user-specific application configuration settings. Platform as a Service (PaaS): The capability offered to a consumer is to deploy applications that the consumer has created or acquired, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and, potentially, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The functionality provided to the consumer is the provision of processing, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control over select networking components (e.g., host firewalls).

[0019] The layout model is as follows: Private Cloud: The cloud infrastructure operates solely for one organization. It can be managed by that organization or a third party and can exist on-premises or off-premises. Community Cloud: The cloud infrastructure is shared by several organizations to support a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by those organizations or by a third party and can exist on-premises or off-premises. Public Cloud: Cloud infrastructure is made available to the public or large industry groups and is owned by organizations that sell cloud services. Hybrid Cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound to each other by standardized or proprietary technologies that allow portability of data and applications (e.g., cloud bursting for load balancing between clouds). Cloud computing environments are service-oriented with a focus on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure, which contains multiple interconnected nodes.

[0020] FIG. 1 illustrates an exemplary cloud computing environment 50. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N, or combinations thereof, communicate. The nodes 10 can communicate with each other. They can be physically or virtually grouped in one or more networks (not shown), such as private, community, public, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, or software-as-a-service without the need for the cloud consumer to maintain resources on a local computing environment. It will be understood that the computing devices 54A-N supported in FIG. 1 are exemplary only, and that the computing node 10 and computing environment 50 can communicate with any type of computerized device over any type of network, or a network-addressable connection (e.g., a web browser), or both.

[0021] Referring now to Figure 2, a set of abstract layers of functionality provided by cloud computing environment 50 (Figure 1) is shown. It should be understood in advance that the components, layers, and functionality shown in Figure 2 are intended to be illustrative only, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functionality are provided:

[0022] The hardware and software layer 60 includes hardware and software components. Examples of the hardware components may include a mainframe 61, a plurality of servers based on a RISC (reduced instruction set computer) architecture 62, a plurality of servers 63, a plurality of blade servers 64, a plurality of storage devices 65, and a network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0023] The visualization layer 70 provides an abstraction layer from which embodiments of virtual entities, described below; virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0024] In one embodiment, the management layer 80 may provide the following functions: A resource provisioning unit 81 provides dynamic acquisition of computing resources and other resources used to perform tasks within the cloud computing environment. A metering and pricing unit 82 provides cost tracking as resources are used within the cloud computing environment and provides billing or charging for the consumption of these resources. In one embodiment, these resources may include application software licenses. A security unit provides identification and authentication of cloud consumers and tasks as well as protection of data and other resources. A user portal unit 83 provides accessibility to the cloud computing environment and system administrators for consumers. A service level management unit 84 provides allocation and management of cloud computing resources to meet required service levels. A service level agreement (SLA) planning and fulfillment unit 85 provides pre-provisioning and acquisition of cloud computing resources as future requests are required according to SLAs.

[0025] The Workload Layer 90 provides examples of functionality for utilizing a cloud computing environment. Examples of workloads and functionality provided by this layer include mapping and navigation 91; software development and lifetime management 92; virtual classroom instructional delivery 93; data analytics processing 94; transaction processing 95; and in the context of an exemplary embodiment of the present invention, accelerators and cloud-based services for fine-grained level visual recognition in mobile augmented reality 96, although it is understood that cloud, non-cloud / local and combined approaches could be utilized. For example, some embodiments reside on the user's smart phone, laptop, or other mobile device 54A, 54C and optionally interact with elements in the cloud.

[0026] One or more embodiments effectively provide a system and method for augmented reality-driven visual recognition. Despite recent advances in augmented reality (AR), most AR user experiences remain primitive and lacking intelligence and automation, which makes user interaction rather unintuitive. AR allows tracking of virtual objects and annotations in physical space through computer vision techniques, but is not intelligent enough to actually recognize the semantics of what is seen. For example, in the technical support domain, AR can recognize a laptop in the form of a point cloud to allow annotations tracked on the laptop, but it does not actually know that the user is looking at a laptop unless the user labels that particular point cloud as a laptop. Also, AR would not know if the laptop's cover is open or closed, nor would it know that the laptop has had its heat sink removed, etc. The lack of semantics severely limits interaction, as a fine-grained level of recognition requires that all interactions be driven by the user by identifying what the user is looking at (e.g., pre-labeling objects, identifying object states, etc.) before the user can have the relevant AR content projected into the user's field of view. To address this gap and provide an enhanced AR user experience, one or more embodiments provide a finer-grained level of visual recognition approach that recognizes not only objects, but also object parts or state change(s). This approach enables an AR application to automatically provide relevant AR instructions in the correct context by detecting state changes of object parts.

[0027] Fine-grained category recognition (e.g., car models, bird species) is an extremely challenging problem since it requires the ability to identify and distinguish subtle visual differences. Most existing work relies on human-annotated datasets that act on bounding boxes to represent relevant features, which is extremely difficult to scale. Weakly supervised part models using convolutional neural networks (CNNs) with category labels greatly improve their capabilities since they do not have a dependency on bounding boxes for data labeling. Such approaches involve two steps: identifying parts by training from positive / negative image patches, and then extracting fine-grained features for recognition. However, these approaches are not directly applicable when dynamic camera movement introduces noise into the captured images. Similarly, Region Proposal Networks (RPNs) do not perform well in complex scenarios such as state recognition for hardware repair because many parts (e.g., RAM sticks, screws) look the same, yet their combinations can represent different states (or even multiple states) of the hardware. Such RPN-based models lack the ability to focus on specific parts of interest. Existing approaches share the same assumption that distinguishable parts are properly captured in the input image, which may not always be the case in practice.

[0028] Prior art techniques assume either input images captured from typically 12 fixed viewing angles around the object (30° azimuth steps), or the use of random images in the form of groups or pairs. Rather than assuming, as in the prior art, that the user knows how and where the input images were captured, one or more embodiments guide the user to capture images from the most informative regions of interest (ROIs) based on the camera pose and a 3D point cloud representing the target object.

[0029] Active recognition typically involves learning a generative model of an object and then selecting a view based on entropy reduction. Recently, the “ShapeNets” framework has been proposed to model objects as voxel grids and to learn a generative model based on convolutional deep brief networks, allowing view synthesis from unseen viewpoints. Frameworks for learning end-to-end policies for active visual characterization and exploring unseen environments for unknown tasks have also been proposed. However, these models typically cannot guarantee completeness of object recognition so that all relevant parts are captured.

[0030] One or more embodiments advantageously provide a solution that takes advantage of AR-specific data, such as real-time generated 3D feature points and camera pose(s), to supplement the images captured by the camera for fine-grained level visual recognition. First, a set of training videos is used and ROIs with appearance changes that characterize different states are learned. Actively tracking the 6DOF camera pose ensures that the camera is held at the correct distance and viewing angle relative to the ROIs, while minimizing occlusion or other noise in the input image of the visual recognition model. To improve the robustness of recognition, one or more embodiments provide a discrete multi-stream convolutional neural network (CNN) in combination with a bidirectional long short-term memory (LSTM), i.e., a discrete CNN-LSTM (DCL) model, to extract not only spatial but also temporal data to predict state changes.

[0031] One or more embodiments address the unique problem of fine-grained visual recognition in a mobile AR setting and actively predict object state changes by combining images, 3D feature points and camera pose data. One or more embodiments generate ROI candidates from merged 3D feature points collected from an AR session and use deep CNN feature representations to automatically extract distinct ROIs that are tracked and retrieved for fine-grained recognition. One or more embodiments use an inventive discrete CNN-LSTM (DCL) model that distributes ROI images over discrete multi-stream CNN branches and aggregates the information with a bidirectional LSTM layer. The weight-sharing multi-stream CNN solves the conflict between high image resolution (and therefore larger model size) required for fine-grained levels and the resulting lower resolution with an increasing number of ROIs. The LSTM layer aggregates visual information in the temporal domain to further enhance prediction stability. In a non-limiting example embodiment, an iOS® application is built using an AR development platform (Apple®'s AR Augmented Reality Development Platform for iOS®) and an end-to-end machine learning platform (the open source TensorFlow Platform) to demonstrate effectiveness using a hardware maintenance application scenario and provide a comprehensive evaluation.

[0032] Augmented reality (AR) is currently used for a variety of purposes, including technical support. AR systems are computerized vision systems that overlay virtual materials onto the real world. Augmented self-assistance involves step-by-step three-dimensional (3D) animated instructions overlaid on an image of the hardware on which the task is to be performed; this is more intuitive and engaging than traditional systems. Traditional systems lack a fine-grained level of visual awareness and typically require human analysis and manual selection of the applicable set of instructions to view. One or more embodiments advantageously recognize the state and repair of an object and automatically provide the correct set of instructions. As defined herein, the "visual state" of an object refers to the visual configuration of the object, e.g., the machine being repaired at a given time; e.g., the characteristics of the object being present, e.g., battery installed / not installed, latched up, or down-present, as opposed to the object itself merely being present / present. In the battery example, the "visual state" is the state of the motherboard as opposed to the battery. In one or more embodiments, this is made possible by a level of visual awareness that has not been available before. See FIG. 3A. It is desired to determine whether the fan 303, or the battery 305, or both, have been removed. Due to the distance from the camera, it is difficult to determine whether the battery has been removed, so it is appropriate to "zoom in" as in FIG. 3C. In FIG. 3B, another possible problem can be seen; the battery 305 is hidden by the wire 307. In FIG. 3D, yet another possible problem can be seen; the battery 305 is hidden by the user's hand 309. Also of note is the CPU 311.

[0033] One or more embodiments advantageously provide a visual recognition system that recognizes the visual condition of a machine being repaired, which may provide, for example, one of the following functions: Automatically presenting corresponding AR instructions. Verify that all prerequisites for moving to the next step are met. Being able to answer the question "How far have I got?!" (knowing the state of the machine after an interrupted repair process).

[0034] One or more embodiments provide a very fine level of recognition of an object. An object can be advantageously viewed from an appropriate viewing angle so that important components are visible to the camera. One or more embodiments can zoom in on a localized area of ​​an image to detect, for example, whether a small screw is installed or not (or other fine level of detail). The object may be obscured by a user's hand or tool (FIG. 3D), which introduces "noise" into the image.

[0035] Additionally, instead of traditional visual recognition scenarios that take well-photographed images as input, the input images in AR scenarios are captured from a dynamically moving camera. One or more embodiments address the challenge of reliable and stable recognition results for partially captured or blurred images caused by fast camera movements.

[0036] Conventional current visual recognition algorithms have not been able to provide good results in such problematic cases. One or more embodiments advantageously provide a framework that allows fine-grained level recognition. With reference to FIG. 4, one or more embodiments include an offline phase 403 and an online phase 405. For the offline phase, due to AR, an object can be scanned by a camera to detect three-dimensional (3D) feature points of the object, and the object's pose, orientation, and camera position can be tracked. In one or more embodiments, the object can be first scanned in the offline phase at 407 to build a 3D model of the object. Meanwhile, a video frame / image of the object is captured at 409. The captured data is used for ROI (Region of Interest) extraction 411. For example, the battery and CPU fan are regions of interest. One or more embodiments use an algorithm to automatically extract such ROIs (offline ROI capture) 413.

[0037] Once the ROIs are captured (offline 413, online 414), one or more embodiments determine the ROI image (offline 415, online 416) - which may include, for example, calculations to determine how to collect thumbnail images in addition to thumbnail concatenation. For example, concatenate multiple images of interest (in a non-limiting example of 541, 9 images in a 3x3 arrangement of 803-1, 803-2, ..., 803-M) (battery, CPU, chip, etc.); the concatenated image can be used for visual recognition 471 during the online phase 405. In one or more embodiments, during the offline phase 403, a deep learning model is trained 419 based on the concatenated images 415. The model is used online in real time for visual recognition 417.

[0038] In one or more embodiments, visual recognition is a two-step process: (i) ROI extraction 411 and (ii) ROI concatenation and visual recognition. One or more embodiments therefore do not feed the entire image to the model for visual recognition.

[0039] FIG. 4 shows the steps involved for example system design, which can be split into offline 403 and online 405 phases. In the offline phase 403, the object is first scanned (e.g., with a moving camera) at 407 to build an object model, which is a 3D point cloud representation of the object, which can be used for repositioning 404. During the scan, the camera pose, detected point cloud, and video frames are collected at 409 to generate ROI candidates. For ROI extraction, the 3D point cloud representation from the object model is downsampled at 411 to generate ROI candidates, which are further filtered at 413 (e.g., (i) too fast camera movement, or occlusion of the region of interest, or both) so that only those ROIs with changes in appearance are selected. Based on the selected ROIs, saved frames are collected to collect ROI images 415, which are used to train 419 the recognition model.

[0040] In the online phase 405, we first detect the object using the object model 404, and then reposition the mobile device relative to the object so that the extracted ROIs stay in the same position relative to the object. Based on these ROIs of the object, we zoom in on each ROI 414 to collect local high-resolution images, which are further processed to obtain images 416 used for visual recognition 417 (online) (images 415 are used for training the model 419 (offline)). Based on the recognized “state” of the object, the corresponding AR instructions are automatically presented.

[0041] FIG. 5 illustrates an alternative view that is discussed herein.

[0042] FIG. 6 shows an exemplary process for ROI generation. First, at 503, the object of interest (here, a laptop computer) 502 is scanned with an AR development platform; one non-limiting example is the ARKit augmented reality development platform for Apple®'s iOS® mobile devices; another is Google®'s ARCore software development toolkit (experienced engineers will be familiar with ARKit, ARCore toolkit, etc.). As seen at 505, this is aggregated into a point cloud of feature points; in essence, a rough representation of the object. Then, at 507, outliers are detected, which are illustrated in red in view 509. At 511, the outliers are removed to obtain view 513, a point cloud representation of the object 502. At 515, a downsizing of the voxel grid is performed to obtain view 517, where each point is the centroid of a small voxel in 3D space. In essence, it is downsampled to a fixed size. Then, at 519, the voxel normals are estimated to obtain view 521. At 523, view 521 is processed to obtain candidate regions of interest 525-1, 525-2, 525-3, 525-4, 525-5, 525-6, and 525-7. Only seven are shown to avoid clutter; however, in general, regions of interest are generated to cover the entire surface of object 502. Typically, there is an ROI for each dot with a salient normal; only seven ROIs are shown to avoid clutter. At 529, the candidate ROIs are projected onto the 2D image to obtain view 531.

[0043] One or more embodiments act to track camera pose in AR to keep the camera focused on regions of interest (ROIs), so noisy background information is filtered out; the results are robust to any device movement.

[0044] Now, referring to FIG. 7, we identify those ROIs that are most discriminative between different object states. We focus on these (distinguishable) ROIs and exclude the noisier ROIs (i.e., those that contain background). The regions of interest are represented by 3D coordinates, which are projected onto a 2D image. At 533, we perform ROI extraction to obtain views 535 of the two ROIs. Then, at 537, we perform image collection to obtain zoomed-in and enlarged images 539 (small regions of interest); an image is typically obtained for each ROI for each state of interest. The collected images are concatenated at 541 as shown in the lower right-hand corner of FIG. 7. A model 545 is trained to classify such images, as shown at 543. During the training process, it is possible to determine which ROIs are most significantly affected for each state change. For example, if one wishes to detect whether the battery has been removed or not, the user is guided to focus on the battery ROI to resolve any confusion regarding the actual state. In one or more embodiments in this context, ROIs refers to all of the ROIs that have a significant impact. It is also known which ROIs have a visual change from one state to another. For example, one or more examples have four states: battery on, battery off, cpu on, cpu off, and two ROIs (battery and cpu). If the system is uncertain about the battery being on / off, then this ROI should be scanned instead of the cpu. Each ROI is associated with a certain state change; in one or more embodiments, this is how the user is guided to move the camera in case of uncertainty. The probability of each state indicates which state is desired to be detected. For example, if the state / probability is state 0: 0.05, state 1: 0.45, state 2: 0.5, it means that the system is confused by states 1 and 2. The corresponding ROIs should then be scanned.

[0045] Now, referring to FIG. 8, in one or more embodiments, at runtime, we take video frames 703 as input; 3D feature points 705; camera pose data 707 (camera orientation and position) as input, and perform a 3D to 2D projection at 709 based on the ROI, while taking images for each ROI and concatenating them together for recognition. Also referring to 708, one or more embodiments capture ROI images only if the voxel is occupied. As seen at 711, we take zoomed local images and view poses for each ROI, which can be concatenated to obtain the recognition model 545. This technique advantageously allows for very fine-grained level image acquisition with high accuracy, by using high resolution images and by considering and collecting only ROIs with changes. One or more embodiments are robust to camera movement and automatically ignore images that are occluded and / or blurred (e.g., due to premature camera movement and / or rotation).

[0046] Thus, one or more embodiments leverage camera pose tracking and 3D feature points, which are "by-products" of AR. Beneficially, one or more embodiments are able to detect changes at a very fine granularity with high accuracy, since highly detailed sub-areas are captured. Such areas are zoomed in and collected during scanning. One or more embodiments are robust to occlusion from a user's hand, tool, etc., since 2D images are captured only when the voxel of interest is occupied. One or more embodiments are robust to camera movement and automatically ignore blurred images.

[0047] FIG. 9 illustrates aspects of the model design. Multiple branches 803-1, 803-2, ..., 803-K are provided for future extraction. These are obtained by separating the ROIs into multiple branches, such that 803-1 contains ROIs zero through eight; 803-2 contains ROIs nine through seventeen; and 803-K contains ROIs 9*(K-1) through N-1 (N=total number of ROIs, numbered from 0 through N-1, and K=number of branches). In some instances, there may be fewer than eight ROIs, e.g., only a single branch 803-1 may be utilized. Putting too many ROIs in one image undesirably compresses the resolution since the input to the CNN is fixed. Therefore, the ROIs are typically distributed across multiple branches; this beneficially saves computational overhead, since updating one ROI image only requires running one branch of feature extraction. For example, the feature extraction process is generally shown at 805, which uses a convolutional neural network(s) 807. At 809, spatial information aggregation is performed by concatenating all extracted features 810, as seen at 811. At 813, temporal information aggregation is performed: for each time step tn, ..., t, spatial information is pooled. In one or more embodiments, a long short-term memory (LSTM) layer 815 is used, which finally provides a fully connected layer 817 for classification 819. Distributing the ROIs into multiple branches, where each ROI has an appropriate resolution, is advantageous in one or more embodiments.

[0048] One or more embodiments provide a method for fusing 2D and 3D frames in AR for fine-grained visual recognition, which is robust to random camera movement and occlusion. One or more embodiments generate ROI candidates from sparse 3D feature points collected and accumulated from an AR session, and use gradient-weighted class activation mapping (Grad-CAM) to extract the most distinguishable ROIs. One or more examples utilize accurate tracking and collection of camera poses for ROI tracking, so that the recognition model can focus on the ROIs no matter how the camera moves (some embodiments perform automatic collection of ROIs according to the camera position). This is more effective and robust compared to region proposal networks. One or more embodiments use a multi-stream CNN with bidirectional LSTM including a neural network with attention mechanism that takes a sequence of ROIs and camera poses as input, and extracts spatial and temporal information for robust visual recognition.

[0049] One or more embodiments can detect very fine-grained levels of change with high accuracy because highly detailed subareas are captured. Such areas are zoomed in and collected during scanning. One or more embodiments are robust to occlusion by a user's hand or other tool because 2D images are captured only when voxels of interest are occupied. One or more embodiments are robust to camera movement and automatically ignore blurred images.

[0050] One or more embodiments automatically identify ROIs by learning from data and taking as input not only images and poses but also 3D feature points. In one or more embodiments, multiple ROIs are captured simultaneously and ROIs from images are automatically tracked and collected using camera poses and projections. One or more embodiments use a multi-stream CNN-LSTM with concatenated images of distributed ROIs as the underlying deep learning model.

[0051] One or more embodiments provide a method for generating augmented reality driven repair guidance, comprising: scanning an object of interest using a mobile device; extracting camera pose and frame information from the scan; extracting frames showing regions of interest defining visual changes in aspects of the object; collecting frames showing regions of interest to include only relevant portions; and automatically detecting occlusion of the target region of interest to activate a deep learning model only when there is a change in reality. Further steps include feeding the collected frames to a deep learning model to detect visual changes in the state of the object; using the detected object state to determine which instruction(s) to provide through AR; and guiding a user to the relevant ROI based on the real-time prediction results.

[0052] In some embodiments, extracting frames showing regions of interest is performed automatically by training a deep learning model based on CNN, which includes: generating candidate regions of interest; and selecting the most relevant regions of interest. In some such cases, collecting frames showing regions of interest to include only relevant portions to increase visual recognition accuracy includes: collecting and concatenating regions of interest from image frames; and data augmentation of distinct regions of interest in the image and their concatenation. In some such cases, the collected frames are fed to a deep learning model to detect visual changes in the state of the object; a multi-stream CNN-LSTM model takes the distributed and concatenated images as input for estimation.

[0053] A skilled technician will therefore recognize that visual recognition in AR can play a key role in enabling the evolution of AR-assisted repair. One or more embodiments provide an approach for fine-grained level active visual recognition in augmented reality, which allows for extremely fine-grained level of detail recognition of complex 3D objects, such as a screw removed from a machine. One non-limiting exemplary implementation uses an iOS®-based mobile app and can be used for hardware maintenance.

[0054] The use of AR tracking for visual recognition is of great relevance to one or more embodiments. One or more examples provide frame filtering including pose clustering and initial sensor-based frame selection. The embodiments can be used for many different applications; for example, AR-assisted repair work (automotive, electronics, self-assembly furniture, etc.); education (crafts, etc.); entertainment, games, etc.

[0055] One or more embodiments are not limited to classifying object categories, however, they can classify fine-grained levels of state. One or more embodiments can handle 3D objects as well as 2D images. One or more embodiments work well with cameras commonly found on mobile devices such as phones, tablets, etc. One or more embodiments recognize 3D objects in video frames and act on camera poses from AR.

[0056] Augmented reality is increasingly being explored as a novel medium for two-way remote collaboration applications to effectively and efficiently guide participants through visual instructions. Users are striving for more natural interaction and automation in augmented reality applications. One or more embodiments provide novel visual recognition techniques to enhance the user's experience. While simple object recognition is often used in augmented reality to this goal, most collaboration tasks are too complex to be satisfied for such recognition algorithms. One or more embodiments provide an active, fine-grained level visual recognition approach for mobile augmented reality that works with 2D video frames, 3D feature points, and camera pose data to detect various states of objects. One non-limiting example application is a mobile application for hardware support that automatically detects the state of the object for which support is sought in order to present the right set of information in the right context. One or more embodiments advantageously overcome the limitations of the prior art, which requires all instructions to be driven by the user due to the lack of semantic, fine-grained level recognition.

[0057] One or more embodiments provide a rich AR user experience, a finer level of visual recognition, i.e., recognizing not only objects but also changes in the state or parts of an object, and usability in a broad range of application scenarios including technical support. One or more embodiments advantageously overcome various challenges in providing such finer level visual recognition capabilities. For example, referring back to the example of FIGS. 3A-3D, a technician is replacing the CPU 311 of the computer. In order to provide relevant AR instructions in the correct context, the visual recognition mechanism would have to be able to detect the change in the computer's parts, e.g., the motherboard. Only if both the battery 305 and the fan 303 are removed from the motherboard would an AR instruction for CPU removal be provided. Compared to traditional visual recognition methods, implementing such state recognition requires addressing the following challenges in AR applications: i) Camera Distance: Depending on the target object or object parts that need to be recognized, machine learning models typically require the camera to change distance relative to the object. For example, in Fig. 3A, to recognize the entire machine in the camera field of view (FoV), the camera needs to move away from the machine, but to recognize the battery, the camera needs to move closer to the machine. This requires dynamically adjusting the camera distance to achieve a fine-grained level of visual recognition. ii) Viewing Angle: The relevant target object parts must be visible to be recognized. This means that the camera should capture the object at a certain angle. Figure 3B shows the battery being viewed from a certain angle hidden by wires, while in Figure 3C the battery is clearly visible. If the proper viewing angle is not maintained, visual recognition will naturally be less accurate. iii) Noisy Input: As shown in FIG. 3D, a common challenge in AR-driven hardware support is that objects may be temporarily occluded by a user's hand or tool. It is also common for input images to be captured by a moving camera, and thus they may be blurred in time. All these factors make the input images for the visual recognition model noisy. One or more embodiments are advantageously robust in handling such noisy inputs, providing reliable and stable results. In addition to these challenges, one or more embodiments are designed to differentiate within the resource and power limitations of mobile devices.

[0058] One or more embodiments enable a machine to mimic human perception and reasoning to detect state changes; embodiments allow a camera to focus on distinct localized areas that change appearance in different states; take care to appropriately adjust the viewing angle to collect images from these localized areas; and make a prediction(s) about the state change only once sufficient visual data has been collected.

[0059] One or more embodiments take advantage of AR-specific data, such as real-time generated 3D feature points and camera pose(s), to supplement the images captured by the camera for fine-grained level visual recognition. One or more embodiments first use a set of training videos to learn regions of interest (ROIs) with appearance changes that characterize different states. One or more embodiments actively track the 6DOF (six degrees of freedom: x, y, z, translation, plus, pitch, yaw, and roll) camera pose to ensure that the camera maintains the correct distance and viewing angle to the ROIs and minimize occlusion or other noise in the input image of the visual recognition model. To improve the robustness of recognition, one or more embodiments combine a discrete multi-stream convolutional neural network (CNN) with a bidirectional long short-term memory (LSTM), i.e., provide a discrete CNN-LSTM (DCL) model to extract not only spatial but also temporal data to predict state changes.

[0060] We address the unique problem of fine-grained visual perception in a mobile AR setting by combining image, 3D feature points and camera pose data to actively predict object state changes.

[0061] ROI candidates are generated from merged 3D feature points collected from the AR session, and then automatically extract distinguishable ROIs using tracked and collected deep CNN feature representations for fine-grained level recognition.

[0062] We present a discrete CNN-LSTM (DCL) model, which distributes the ROI images across discrete multi-stream CNN branches and aggregates the information in a bidirectional LSTM layer. The weight-sharing multi-stream CNN solves the conflict between high image resolution, and therefore larger model size, required for fine-grained levels, and the degradation of resolution with increasing number of ROIs. The LSTM layer aggregates visual information in the temporal domain to further enhance prediction stability.

[0063] Providing mobile (e.g., iOS®) applications using large-scale machine learning systems such as ARKit and TensorFlow, for example in hardware maintenance application scenarios.

[0064] The acronym AFVR is used herein to refer to Active Fine-Grained Level of Visual Recognition. It is useful to consider basic AR concepts that are relevant to one or more embodiments.

[0065] Object Model. An object model is a point cloud, or 3D features, that represents a physical object. It is constructed from images taken from different angles of the object in a view. During an AR session, the object model is used by the mobile device to reposition itself, i.e., understand its relative position and orientation with respect to the physical object. Repositioning of the device is highly relevant to one or more embodiments since it determines where the AR content is projected in relation to the object. One or more embodiments use the object model not only for repositioning but also to determine the ROIs.

[0066] Camera pose. Camera pose refers to the position and orientation of a mobile device in 3D space. AR software development kits (SDKs) such as ARKit and ARCore track camera pose using motion sensing in combination with computer vision analysis of the scene, acting on visual-inertial distance. Camera pose is represented as a 4x4 matrix M, which describes the rotation, scale, and translation of the device coordinates.

[0067] Feature points. Feature points represent salient features detected on an image. The feature points collected from different view angles form an object model, or point cloud, which is used for AR tracking. The feature points for each frame are denoted by P={p0, p1, ..., p N}, where p i ={x i , y i , z i} is the position of point i in the 3D space, and N is the total number of points.

[0068] In one or more embodiments, the first step of AFVR is to identify regions of interest (ROIs) that represent state changes of the object. The ROIs are acted upon to keep the camera focused to capture appropriate video frames for state recognition. Finally, noise in the frames is filtered out to improve the robustness of the recognition. Figure 5 shows an exemplary system design, which includes both offline and online components.

[0069] In the offline phase 403, we first harvest data from AR remote sessions via crowdsourcing 599 or let expert users scan objects with their mobile devices (as 597) and build relevant object models (at 595) - 3D point clouds that are representations of the object in different states. We also collect camera poses, corresponding feature points and video frames. Then, an ROI extraction step 593 generates a set of ROIs based on the video frames collected in different states. These ROIs will determine what images should be generated from the video frames to recognize the object states.

[0070] In the online phase 405, the AFVR first detects 591 an object and repositions the mobile device relative to the object using the object model. The ROIs 589 identified in the offline phase are also mapped onto the object as described herein. Images of these ROIs are then collected to store only the relevant regions of the ROIs 587 and further processed 585 to train a model for state recognition. During real-time recognition 583, the mobile app instructs the user (at 581) to position the camera at the correct distance and viewing angle relative to the object and applies the trained recognition model to predict the current state. Based on the predicted state, an applicable object model is automatically selected for AR tracking and corresponding AR instructions are provided accordingly.

[0071] Also, referring to FIG. 10, it is shown how pose-independent coarse-grained recognition selects an accurate object model for repositioning, which enables fine-grained level recognition. Coarse-grained recognition 1003 on multiple video frames (1001-1, 1001-2, 1001-3; any suitable number of frames can be used) is performed. Decision block 1005 determines whether there has been a state change. If not (NO branch), proceed to fine-grained level recognition 1011, present AR instructions to the user at 1013, and the user manipulates the device at 1015 while additional frames are captured. On the other hand, if there has been a state change (YES branch of decision block 1005), load a new object model at 1007, reposition the device at 1009, and then continue with fine-grained level recognition 1011.

[0072] Constructing an object model: To build an object model, ARKit's (or similar) scanner functionality is used to scan a physical object and extract feature points to generate a 3D point cloud. Feature points extracted from video frames are accumulated along the way from different view angles. Note that there is a trade-off between accuracy and computational overhead of AR tracking, determined by the density of feature points in the point cloud. In one or more embodiments, a relatively sparse AR tracker and a denser point cloud for ROI identification are operated on as described herein.

[0073] In one or more embodiments, sparse and dense point clouds are constructed separately for a physical object in different states. This can be achieved, for example, by performing separate scans in each state if the user knows the different states of the object in advance. Point clouds representing different states can also be obtained implicitly from an AR-driven remote assistance session if the user explicitly marks state changes verbally or manually during the session. An exemplary inventive point cloud generation algorithm takes as input images corresponding to each implicit state and generates the associated point cloud.

[0074] In implementation, the generated point cloud may have many noisy feature points, for example due to imprecision in depth estimation. One or more embodiments further filter the feature points by removing “non-smooth” outliers from the neighborhood of each feature point. In particular, for each point p i For all its k neighboring points q j Average distance to (j=1, 2, ..., k):

[0075]

number

[0076] and standard deviation

[0077]

number

[0078]

number

[0079] All points with distance d>μ+ασ are considered outliers and are therefore removed, where α is a parameter that controls the “smoothness” of the point cloud surface. In one or more embodiments, we set h=50 and α=1.0. As shown in FIG. 6, we first detect the ones marked in red in 509, and then a clean point cloud 513 is derived.

[0080] After this step, we obtain point clouds that robustly represent the object in different states. For each state, a dense point cloud with more feature points is generated for ROI identification, and at the same time, a downsampled sparse point cloud is generated for efficient tracking. The point clouds generated in different states can be easily aligned to the same 3D coordinate system because they use the same sparse point cloud for relocation. ARKit relocation has a certain tolerance to appearance or shape changes, so the relocation operation works well as long as the object changes are partial or small.

[0081] ROI Identification: An ROI is a segment in 3D space where the physical appearance of an object changes due to state changes. To visually recognize changes in an ROI, the ROI is projected onto a region of a 2D image acquired from a fixed viewing angle of the object. The simplest way to recognize an ROI is to rely on human knowledge: an expert user can illustrate the boundaries of ROIs on a given image. This is in practice redundant and easily leads to inconsistencies in ROI or state determination due to different human interpretations. Therefore, one or more embodiments provide an approach for automatically identifying ROIs, given a set of images, and labeled point clouds with different states, as discussed herein.

[0082] Voxelization: In one or more embodiments, 3D space is segmented into voxels of a fixed size. An ROI contains one or many voxels, which may contain several feature points (or none at all). To identify ROIs, a consistent method for projecting a voxel into a 2D image is appropriate. For that purpose, an anchor point is defined for a voxel as the centroid of all feature points contained within this voxel. When projecting a voxel, this anchor point is first projected into the 2D image. A square region of width w centered around this anchor is collected from the image to represent the corresponding voxel. This ensures that the projected image contains sufficient visual details of the object surface. View 517 shows the anchor points for all voxels of the sample object.

[0083] To ensure that sufficient visual detail is collected for a voxel, ensure that the object is visible within the camera field of view. This is made possible by estimating the normal of the anchor point with respect to the object surface. The normal vector is estimated by calculating the normal of a plane that is tangent to the object surface at the anchor point. This can be achieved by least squares plane fitting by using a library such as the Point Cloud Library (PCL). An example of an estimated normal vector is shown in view 521.

[0084] ROI Image Selection: For a given camera-captured image of an object, collect ROI candidate images that provide input for object state recognition. The ROI candidate images are square segments of width w that are collected around the voxel anchors in {x, y, z}. Project the voxel anchors and all vertices of the ROI candidate onto the camera-captured image and select the minimum bounding rectangle that covers the projected ROI as the collection area. Given a camera pose (transformation matrix M camera ) and a point in 3D space, P world For this point, we convert it to camera coordinates as follows:

[0085]

number

[0086] where P camera is the projected point in camera coordinates. We use perspective projection to project a point in camera coordinates onto the image plane by simply dividing the x and y coordinates of the point by its z coordinate:

[0087]

number

[0088] Then we transform the 2D points in image space into raster space, where the pixel coordinates are:

[0089]

number

[0090] where W, H are the width and height of the canvas in raster space, and P′ norm x,P′ norm ·y are normalized coordinates in raster space, which are further multiplied by the resolution of the image frame to obtain pixel coordinates. An ROI is in the FoV if and only if all four projected vertices are visible. Images for each visible ROI are collected from the raw full resolution image frame and resized to a fixed resolution. Multiple ROIs can be collected simultaneously as long as they are in the FoV and visible to the camera in one frame. A few samples of candidate ROI images after the collection step has been applied are shown in 525-1 to 525-7.

[0091] Then, from these ROI candidates, select the one that can best differentiate the object states. One or more embodiments use gradient weighted class activation mapping (Grad-CAM) to perform ROI image selection. Images collected for each object state are available from different viewing angles. Collect the respective images to obtain ROI candidate images. Sort the collected images according to the location of the corresponding voxels and concatenate them into a single image, which are resized to a fixed resolution (e.g., 224×224) as input for GRAD-CAM, as shown in the left-most of the three images 541. For each state, a subset of these candidate images reflects the appearance changes of the object.

[0092] Deeper representations in convolutional neural networks (CNNs) capture higher-level visual features. State-specific information in the input image (i.e., voxels with appearance changes) is usually captured in the last few convolutional layers with higher-level semantic and spatial information. Grad-CAM uses gradient information flowing into the last convolutional layers of CNNs to understand the importance of each candidate image (other methods such as SIFT and ORB could be used in other embodiments, however, in experiments we found these techniques to be ineffective in identifying ROIs because they search for similarities rather than identifying subtle visual changes). The goal of ROI selection is to obtain a localization map of width u and height v for any class c that distinguishes the classes.

[0093]

number

[0094] For this purpose, for the feature maps Ak of ​​the convolutional layer, we define the class probabilities y c The gradient of, i.e.,

[0095]

number

[0096] In this way, the weight α c k Global average pooling,

[0097]

number

[0098]

number

[0099] This gives a coarse heatmap of the same size as the convolutional feature map (7x7 in the case of the last convolutional layer of ResNet50). The heatmap is then normalized to 0-1 and resized to the size of the input image, which can be visualized by overlaying it on the input image, as shown in 541.

[0100] ResNet50, pre-trained on ImageNet, is used and fine-tuned in one or more embodiments as a backbone model to classify the concatenated images from all states and then generate a Grad-CAM map. The Grad-CAM map directly indicates the importance of each candidate image for characterizing a particular state. The left-most image 541 shows a concatenated image of a subset of ROI candidate images collected for a computer motherboard. In this example, the object has three states: default, battery removed, and CPU removed. The collected image showing battery removed is the top right corner of the image concatenation, while that of CPU removed is in the center of the concatenation. The center one of images 541 shows a Grad-CAM heatmap overlaid on the image, precisely showing that the two collected images mentioned above are the best ROI images for differentiating these three states.

[0101] One or more embodiments build a superset of the top 3 ROI candidates for each state selected from Grad-CAM and remove duplicates, and this new set of ROIs is used for training and inferring the final visual recognition model during the online phase.

[0102] Visual Recognition Model: Using the concatenated ROI image as input, we now develop a CNN model to predict the class of the object state. For simple scenarios where an object has a small number of ROIs (e.g., a 6x6 concatenated ROI image), lightweight, shallow max-pooling, and batch normalization are sufficient. As application scenarios become more complex, more ROIs become appropriate to cover all states. In such cases, concatenating all ROI images in a naive way can be problematic: the resolution for each ROI decreases as the number of ROIs increases because the resolution of the concatenated image is fixed, which leads to a loss of visual details. If the resolution of each ROI is maintained, the resolution of the concatenated image increases rapidly and becomes inappropriate as an input to a simple CNN model.

[0103] To address this challenge, one or more embodiments provide a discrete CNN-LSTM model, which includes discrete multi-branch visual feature extraction, followed by a layer of spatial and temporal information aggregation for final classification. FIG. 9 shows the architecture of the DCL model. The multi-branch CNN extracts spatial features from the concatenated ROI image. The LSTM layer further aggregates the temporal information for stable prediction. The feature extraction 805 includes multiple feature extraction branches, each of which takes the concatenated ROI image as input (including up to 9 ROIs). One or more embodiments use a pre-trained ResNet50 on ImageNet as a base model for feature extraction, but remove the last fully connected layer of the model, thus obtaining a 2048-dimensional feature vector for each input image, which is then fed into the pooling layer. In one or more embodiments, the ResNet50 interface operates only if there is an update for the concatenated input image. The image features extracted for other unchanged branches are reused. In this way, we minimize the computational overhead in feature extraction, thereby significantly reducing the computational overhead and power consumption when the model is implemented on mobile devices.

[0104] One or more embodiments concatenate all extracted image features after a pooling layer into a single feature vector, which represents all recognizable visual information for state recognition. Instead of feeding this feature vector to a fully connected deep neural network, one or more embodiments use LSTM to model the temporal relationship between states: the temporal sequence of the feature vector is fed into an LSTM layer, which predicts the state distribution of the object. As discussed herein, by working with the temporal relationship between states, the prediction accuracy of the model is significantly improved.

[0105] Active visual recognition: A visual recognition model can be trained offline using pre-collected labeled data. When building a mobile application that detects the state of an object, we implement the trained visual recognition model and also consider two or more design aspects: repositioning and capturing ROI.

[0106] Repositioning: Repositioning ensures that the camera pose is tracked consistently across an AR session with respect to fixed object coordinates. This concerns both capturing the correct ROIs and presenting the AR annotations in the correct position relative to the object. One or more embodiments use the sparse point cloud acquired during object model generation for repositioning to work with ARKit's object tracking capabilities. In a given AR application, there are two possible state change scenarios: 1) the state change involves a small appearance change, so the same sparse point cloud can be used for tracking; 2) the state change involves a large appearance change, so a new point cloud needs to be loaded and the AR session repositions. One or more embodiments minimize repositioning to reduce tracking inaccuracies that occur during an AR session. To this end, for example, a ResNet50 that uses real-time images acquired from the object is applied to determine the saliency of the state change and triggers repositioning only if it is necessary.

[0107] ROI Capture: One related aspect involves capturing ROIs as segments of images collected from a camera view. To capture ROI images in a robust, online manner, one or more embodiments further address the following issues: occlusion detection, ROI image updates, and feedback to device movements.

[0108] Occlusion detection. An ROI may only be visible to the camera from certain viewing angles, otherwise it may be occluded by other objects in the scene. Figure 11 shows such an example. Camera pose 1101 obtains a normal capture of the ROI, but with the camera at view 1103, the ROI is occluded from the camera's viewpoint.

[0109] A simple approach is to check if enough 3D feature points for an ROI are detected in each frame to determine if the ROI is occluded. However, due to computational overhead, feature points are typically not generated in real-time for every frame. It is of no use for an experienced technician to be familiar with the way feature points are generated in ARKit: as a mobile device moves around, the device motion is updated at high frequency, capturing small displacements. The feature points are then triangulated to infer their 3D position, which is also used by ARKit to calibrate the camera pose. Thus, there is a delay between the detected feature points and the current frame. As the device moves from a “visible pose” to an “invisible pose” (e.g., from pose 1101 to 1103, the feature detected in 1101 is also present in 1103 for a short period of time). To address this delay, in addition to the feature points detected in the corresponding time interval, the past 5 consecutive frames are examined to determine if the ROI is occluded.

[0110] To determine occlusion for a given ROI, three conditions must be met: i) Within the camera's FoV: The projected ROI must be present in the current video frame so that it is ready to be updated. ii) Not enough feature points detected: Enough feature points need to be detected for the ROI capture. If not enough feature points (e.g., >10) are detected for a short period of time, such as 3 seconds, it indicates that occlusion has occurred. One or more embodiments use feature points accumulated over a short period of time to avoid false positives, since the feature points are too sparse even when the ROI is not occluded. iii) Sufficient device movement: Since feature points rely on triangulation, which requires device movement, when a mobile device is stationary, the feature points in the frame are either unchanged or no feature points are generated at all. Without enough feature points, it is typically not possible to discuss whether the ROI is occluded or not.

[0111] ROI Update. When predicting an object online (i.e., while using an AR application), visual recognition uses input images from all ROIs except those that are not visually changing. Meanwhile, as the camera moves around the object, not all of the ROIs are visible by the camera, either because some ROIs are outside the camera's FoV or because they are hidden. There are two types of occlusion: permanent occlusion means that the ROI is permanently hidden by other objects in the object's state (a fan is covering the CPU), and temporal occlusion occurs temporarily due to an inappropriate viewing angle or by other objects such as the user's hand. For a particular ROI, if it is detected continuously for a long time (one or more embodiments empirically set to 5 seconds, but other embodiments may differ) when the device is actively moved by the user and at the same time the ROI is within the FoV, the occlusion is classified as permanent. In such cases, the image for this ROI is reset to zero for consistency (the "fan on" state in FIG. 3A can occur either before or after the CPU is captured, depending on whether the machine is being assembled or disassembled). Similarly, when a component is removed, we reset the associated ROIs. This is detected when the ROIs are within the FoV but are not detected for a period of time when they are not occluded.

[0112] If transient occlusion is detected or the ROIs are out of FoV, we assume that there is no visual change. Therefore, during model prediction, the corresponding feature extraction branch in FIG. 9 is not activated, but the previously extracted vectors are reused to minimize unnecessary computations on the mobile device and prevent the device from overheating. Usually, users naturally see changes in objects through the camera where the AR annotations are displayed, and therefore such changes are passively captured. In rare cases, it is possible for a user to remove a hardware part that has alerted its visual state, but if it is occluded or out of FoV, this will not be considered as a state change and therefore the next set of instructions will not be provided to the user. Thus, the exemplary app recommends the user to move the camera so that the part of the object that triggered the alert is visible, so that the ROIs are effectively updated, enabling reliable and accurate recognition results.

[0113] When a user holds the camera over an object, the captured video frames may have varying quality (e.g., blurred images due to fast movement, unclear images due to viewing angle, or large distance between the camera and the object). To enable robust recognition, one or more embodiments further filter out the ROI images to extract the highest quality image set for the visual recognition model. As shown in FIG. 12, for each ROI, the device's translation and rotation speed are

[0114]

number

[0115] We keep track of the angle α between the camera normal and the ROI normal, and the distance d from the camera to the ROI. p , y p , z p , n xp , n yp , n zp}, where {x p , y p , z p} is the camera position, and n hat ={n xp , n yp , n zp} is the camera normal, and ROI{x r , y r , z r , n xr , n yr , n zr , w r} and the distance from the camera to the ROI is:

[0116]

number

[0117]

number

[0118] The camera movement / rotation speed is determined by the frame rate. t and p t-ΔT The absolute joint translation / rotation velocity is estimated at the camera pose, where the object is captured at a pose with ΔT before the

[0119]

number

[0120] The translation and rotation velocities are normalized to balance their contribution to the combined velocity before calculation. One or more embodiments keep a queue of 5 input images per ROI, and these are updated every 300 ms when a suitable ROI is captured by popping out the first ROI and pushing in a new ROI. 5 images is appropriate in one or more embodiments because it only requires a small time window of 1.5 s, which minimizes the impact on the image captured if the ROI moves out of the FoV. One or more examples convert the ROI images into

[0121]

number

[0122] where δ v is a threshold value, and the image is not blurred if the device moves within this threshold (these are our empirical values ​​over a wide range; δ v (,can be different for different lighting conditions, for example.) To select the optimal ROI, we choose the one with the smallest α, since it affects the performance of most of the suitable ROIs in our experiments. Without such cueing and selection, the last update for the ROI usually occurs when it is about to go out of the FoV, which is not optimal.

[0123] Device movement feedback. To ensure good quality ROI capture, the device's camera must be positioned properly. Feedback can be provided through the ARapp to guide the user to position the camera so that it is in the correct position and that the user does not move the camera too fast or too far from the ROIs. Feedback instructions are shown, for example, only if the ROI images are not captured properly. In particular, if the ROIs are not updated for a time period (e.g. 10s), this means that they are not visible from the right angle or distance, so appropriate instructions are displayed asking the user to move the mobile device. On the other hand, if the recognition result is unclear, the user is instructed to move the device to collect more ROI images until the object condition is not unclear, which is related to such an unclear condition from the ROI identification.

[0124] Experimental Implementation: A non-exclusive implementation of AFVR includes two parts: an offline model training program running on one or more servers, and an online mobile ARapp that operates on the extracted ROIs and the trained model to perform fine-grained level visual recognition.

[0125] Offline Model Training

[0126] Training data generation. ROI collection during training data collection generates multiple images for each ROI under each object condition, which are concatenated as training images for model training. To minimize the training data collection effort, scanning an object only takes a few minutes. Then, we augment the collected data in two further steps.

[0127] (i) Data Augmentation. First, the ROI image is augmented using existing approaches such as random rotation, shifting, and truncation. Random rotation of the ROI image allows for more robust recognition performance when the object is viewed at random angles. Also, the ROI image is shifted and truncated because the captured ROI image may be non-centered and non-centered in a variety of different ways due to offsets in ROI tracking caused by device pose estimation errors. With such augmentation, we position the image for each ROI under each object condition.

[0128] (ii) Image Concatenation. To generate an image that can be consumed for model training, the individual images for each ROI should be concatenated. These ROI images capture the information of the corresponding ROIs, and since they can be captured at different times, they are therefore semi-independent of each other. To augment the concatenated images, randomly draw images from each ROI image set and concatenate them as new training samples. This significantly increases the amount of training data and also captures a diverse combination of ROI images.

[0129] One or more embodiments use the same technique to generate training data for both ROI extraction and model training for fine-grained level visual recognition.

[0130] Model Training. To collect training data, we scan the object in each state for about 2 minutes. Note that this is a one-time effort to build a model for a new object. For evaluation purposes, the training program was run offline on a PC with a GTX1080. Keras API with TensorFlow backend is used in one or more embodiments for CNN construction and training. After training, the visual recognition model is converted to CoreML format to run on the iOS platform.

[0131] Online Mobile App

[0132] We developed an experimental iOS-based mobile application for real-time testing and evaluation. ARKit is used as the AR SDK for repositioning, tracking the camera pose, and delivering the AR experience. The CoreML framework is used to perform on-device machine learning estimation, and schedule tasks for real-time recognition and stabilization of results over multiple trials.

[0133] Cognitive Scheduling. Schedule a timer to check if ROIs are updated every 300ms, and run the CNN feature extraction branch whose input image has been updated. Run spatial and temporal information aggregation and layer classification once per second whenever any CNN feature update is triggered. Reduce unnecessary computation by updating each branch only when the ROI changes.

[0134] Stabilizing Prediction. Furthermore, to stabilize the state prediction result, especially during the transition from one state to another, we apply the rolling average prediction method. We buffer a queue of the latest N prediction results and output the final result only if one predicted state has appeared N-1 times out of N.

[0135] EVALUATION: The AFVR algorithm is evaluated from a number of aspects: recognition accuracy and impact factor, comparison with existing approaches, and resource overhead.

[0136] Recognition Accuracy and Impact Factor: To validate the overall system design, we first evaluate the performance of a simple lightweight CNN model (LW model) with ROI capture mechanism, followed by a comparison with DCL and VGG-based models. The LW model is a two-layer CNN with pooling and fully connected layers for classification.

[0137] State Recognition Accuracy: Consider the task of replacing the CPU of a desktop machine, which involves six steps (e.g., remove screws / fans, unplug wires, open the latch of the CPU, etc.). The machine is scanned under each state for 2 min as training data and another 1 min for testing. Video frames are sampled every 300 ms, so 1 min / s data collection collects 200 image frames. ROIs are extracted and the LW model is trained for evaluation. Figure 13 shows the resulting confusion matrix. The recognition accuracy is close to or above 95% for most states, except state 3, which is a misrecognition of a similar state. This state is more easily confused with a similar state because the visual difference is very small: the fan collector hides the depths inside the machine, and it is only visible from a very limited number of viewing angles. Note that these results are based on individual recognition results. We found that the error results were usually not similar to each other; therefore, we could improve the accuracy and stability with a larger number of trials, which will be evaluated later. Nevertheless, the overall results show that AFVR is robust enough for practical use.

[0138] Factors that affect model accuracy

[0139] Data Augmentation. We evaluated how data augmentation (DA) can improve performance by arranging training images, especially when the training data is limited. Figure 14 shows the recognition accuracy under different amounts of training samples, from 300 to 1500. It can be seen that data augmentation significantly improves the accuracy, especially when the training samples are very limited (e.g., <900). As the size increases, the accuracy performance stabilizes both with and without DA, but is still slightly higher with DA, indicating an improvement in model generation with DA.

[0140] ROI size. The ROI size (length / width of the rectangle) is relevant to achieve the best recognition performance. A smaller ROI size allows a more focused and therefore more fine-grained level of recognition of parts of the object, but also results in a smaller coverage area, which may miss distinguishable parts due to offsets caused by device movement. In such cases, it generates larger errors. A larger ROI size allows for a larger coverage and therefore reduces the chance of missing distinguishable parts, but at the same time, it also captures more background "noise" and reduces the resolution of the important areas for state recognition. We evaluated the effect of different ROI sizes on the recognition accuracy. Figure 15 shows the accuracy obtained with ROIs from 1 to 9 cm. We collected data with ROI sizes of 1, 3, 5, 7, and 9 cm. Experiments show that 5 cm is optimal since it achieved the best accuracy. Depending on the actual size of the object, the optimal ROI size will be different.

[0141] Recognition stability. It is appropriate to have stable and accurate recognition results, since any inaccurate recognition will trigger animations or instructions that are not relevant to the AR application. This will not only confuse the user, but will also confuse them, resulting in a poor user experience. We evaluated the error rate (i.e., inaccurate state recognition) and latency when multiple recognition attempts are made for each cycle. The recognition attempts occur, for example, every 300 ms, so judging one out of the multiple attempts is fast enough without causing any obvious delay to the user. To indicate a state result, at least N-1 attempts must show the same result within N attempts in a cycle. Figure 16 shows that more attempts decrease the error rate more rapidly at the expense of a large latency. This is because more attempts give more stable results, and therefore reduce erroneous results. With 5 attempts, the error rate is less than 0.5%. We choose 5 attempts for each recognition cycle to balance stability and latency. Note that in real-world use cases, the state of the object does not change frequently and a latency of 1-2 seconds can be ignored.

[0142] Comparison with existing approaches

[0143] Accuracy Comparison. We compared the performance of LW, DCL model and a representative object recognition model VGG16. Since ROI capture is a related component of AFVR, we also compared the performance of all models on ROI capture and raw image frames. We fixed all layers except the last two fully connected layers while fine-tuning and training VGG16 with pre-trained weights from ImageNet. Figure 17 shows the results. On ROI capture, all three models achieve high accuracy (>97%), with the DCL model having the highest accuracy of 99.87%. The VGG-based model also has an accuracy of 97.74%, which indicates the effectiveness of the features extracted by VGG16. The results on ROI capture are significantly higher than those using raw image frames as input, which achieve only 50% accuracy. This is mainly due to the lack of occlusion and fine granularity. When the changed parts are occluded, the images may look the same in some states, thus generating a lot of noise in the training and test data. The VGG-based model has the highest accuracy among the three using raw image frames, and is even better than the DCL model, probably because the DCL model takes a large number of “noisy” frames as input, which makes it more difficult to converge.

[0144] Effect of the number of ROIs. As the number of ROIs increases, the resolution of the concatenated image increases if the resolution of each ROI remains the same. If the resolution of the concatenated image is specified, the resolution of each ROI image decreases. Images with too large a resolution may need to be reshaped to fit a model such as VGG. As we collect new data for more complex objects, we manually add more ROIs. Figure 18 shows the results for 8, 16, and 32 ROIs. As the number of ROIs increases, the accuracy of the VGG model drops to <75% due to resizing. LW performs better than VGG at the cost of increasing model size. The DCL model performs best and shows only a slight decrease in accuracy.

[0145] Comparison of model size. The size of the model is relevant for anything that needs to run efficiently, especially on resource-constrained mobile devices. Figure 19 shows the number of parameters for the three methods as the number of ROIs increases. The VGG-based model has a constant number of 14.85M because the image is resized to a fixed resolution of 224x224. The DCL model leverages VGG for feature extraction and therefore has more parameters than VGG. The parameters increase by ~1.6M for every additional 9 ROIs (assuming the input image for each branch contains 9 ROIs). The LW model has the fewest parameters when the ROIs are smaller than 25, and then increases significantly faster than the DCL model. Thus, DCL is more suitable for complex objects.

[0146] Results on Additional Objects: To illustrate generalization, we conducted experiments on five objects, including more complex machines such as servers. Figure 20 shows the box plots of the accuracy of both LW and DCL: both of them are above 95%. The DCL model has a higher median accuracy, above 98%.

[0147] Resource overhead: Resource consumption is related to good user experience and usability. We measured AFVR resource overhead on an iPhone 8. We compared these measures when the mobile app was running in different modes: AR only, AR with LW model estimation, and AR with DCL model estimation. The table in Figure 21 shows the results of a typical measurement for comparison. Visual recognition slightly increases CPU by about 1-1.5% and consumes 30+MB of memory. With all the overhead, AFVR can easily achieve 60fps and does not overheat at all for continuous use.

[0148] Model training overhead. Model training overhead includes two aspects: data collection overhead and computational overhead. For data collection, expert users can scan the object under each condition (each takes about 1-2 mins) or work with crowd-sourced data from remote assistance sessions. In its current state, crowd-sourced data is noisy and requires manual pre-processing. For computational overhead, all data collected on the mobile device is a “by-product” of AR and therefore generates minimal additional overhead. Model training is also lightweight in the backend due to the relatively small amount of data for a given object.

[0149] More Dedicated Models. The exemplary DCL model, which uses VGG16 with pre-trained weights from ImageNet as a base model, gives reasonable results. Training the base model on domain-specific datasets can further improve accuracy and efficiency. Additionally, more dedicated neural networks, such as spatio-temporal convolutional networks, can be used to improve performance.

[0150] Accordingly, one or more embodiments provide an Active Fine-Grained Visual Recognition algorithm that combines 2D video frames and 3D feature points in AR to recognize changes in the visual state of a 3D object even when the changes are minute. One or more embodiments provide reliable and stable visual recognition results that are robust against random camera movements and are implemented through a mobile app focused on iOS-based AR-assisted hardware maintenance.

[0151] One or more embodiments focus on a fine-grained level of visual state, especially in AR. One or more embodiments do not require pre-scanning and pre-labeling each hardware component, and then use some standard pixel-level computer matching algorithms to detect the hardware components. Pre-scanning or pre-labeling individual pieces of hardware is not always possible, since they cannot be separated from the motherboard. One or more embodiments provide a fine-grained level of visual state detection (as opposed to component recognition), which allows tracking to be maintained even with the smallest changes in the scene (such as removing a small screw or lifting a tiny lever). This level of recognition is not possible with 3D point cloud-based object or part recognition, and uses novel deep learning techniques disclosed herein. This includes working with images, 3D feature points, and camera pose in AR.

[0152] One or more embodiments provide an algorithm for visual recognition that recognizes and understands the current AR scene (i.e., cover open vs. cover closed, lever up vs. lever down, screw in vs. screw out, etc.) and automatically presents an applicable set of commands, even though not all commands match every machine state (e.g., a person cannot remove the fan unless the person opens the top cover of the machine). To solve the state recognition problem, one or more embodiments provide a series of deep learning models and techniques that interact with images, 3D features, and camera poses in AR.

[0153] One or more embodiments include a series of specific deep learning techniques for fine-grained level visual state detection, such as automatic ROI identification, ROI collection, and occlusion detection using 3D feature points, that analyze the visual appearance of the ATM and provide targeted instructions on the ATM state.

[0154] One or more embodiments provide a fine-grained level of object state recognition, which requires analyzing a current scene and predicting the state of an object. For example, traditional part recognition allows detecting a CPU cover in a scene as a human would see a motherboard, but one or more embodiments detect whether the CPU cover is open or closed, or its lever is lifted up or down, essentially very subtle differences in the main scene that typical part detection algorithms cannot distinguish. Detecting the visual state of an object (along with subtle changes in the visual state) is a very different task compared to detecting parts of an object. Furthermore, one or more embodiments provide a series of techniques such as automatic ROI identification, ROI collection, occlusion detection using 3D features, etc.

[0155] Previous work requires manual identification of regions of interest (making it difficult to generalize) and does not support occlusion detection (leading to inefficiencies in running deep learning for casual estimation even when the target region is occluded, which may result in overheating of the mobile device). Previous work only takes images and camera pose(s) as input, and captures ROIs one by one, which are fed to a multi-stream CNN for recognition. One or more embodiments train the ROIs automatically (without manual effort) through a deep learning model, operates on a temporal sequence of input images to ensure higher accuracy, and supports occlusion detection for the detected 3D feature points in real time. Finally, one or more embodiments capture multiple ROIs simultaneously and use an improved deep learning model called a multi-stream CNN-LSTM model for recognition.

[0156] Again, one or more embodiments allow for detection of the visual state of an object (cover on vs. off, DIMM on vs. off, CPU lever up vs. down, etc.). One or more embodiments present a series of image ROI capture algorithms and neural networks for object state recognition, which includes all of the detected visual appearances of an object, which is more complex than detecting parts of an object.

[0157] One or more embodiments operate on a mobile device with a normal camera (no special equipment is required - can be accomplished without using a depth-sensing camera) and provide a fine-grained level of object state detection. In the context of the technical support domain, conventional algorithms would enable an AR system to recognize that what it sees is a laptop, but one or more embodiments help detect the state of the laptop (e.g., CPU lever down) and enable the AR system to present the correct command (e.g., lift up the CPU cover). Knowing that the object is a laptop (i.e., a high level of object recognition) is not sufficient to present the correct level of repair instructions, and a finer level of detection granularity is effectively provided by one or more embodiments.

[0158] Again, one or more embodiments provide a method for fine-grained level object state recognition in AR, fully incorporating the effects of AR outputs such as camera pose information and 3D object models. Additionally, one or more embodiments address the problem of occlusion to increase the efficiency and accuracy of deep learning models with 3D feature points detected from AR. With regard to active view selection policies, one or more embodiments can identify which ROIs need to be scanned and guide the user to move the camera within the AR scene.

[0159] Rather than traditional general-purpose classification, one or more embodiments provide a fine-grained level of visual recognition, where distinct regions are only seen from specific viewing angles. One or more embodiments do not need to acquire a fixed number of images, but automatically capture relevant ROIs through camera pose tracking and 3D-2D projection. One or more embodiments apply an LSTM layer to aggregate temporal information.

[0160] In view of the preceding discussion, in general terms, an exemplary method according to aspects of the present invention includes obtaining access to a 3D point cloud representation of the object 502, including the pose of a scanning digital camera (e.g., in the phone 569) and corresponding video frames; this step can be accomplished, for example, by scanning the object with a digital camera coupled to a processor and collecting the pose, corresponding video frames, and corresponding feature points. In one or more embodiments, the poses / points / frames are advantageously a by-product of AR. It is noteworthy that up until now, only images have typically been relied upon for visual recognition. One or more embodiments, in contrast, provide a medium for embedding the results of the model in an augmented reality, thereby generating poses, feature points, frames, etc., within the AR environment. One or more embodiments, in contrast, provide a medium for embedding the results of the model in an augmented reality, thereby generating poses, feature points, frames, etc., within the AR environment. One or more embodiments generate novel models that act on these AR replicas. Referring to ROI capture 589 in the online phase 405 - camera pose can be utilized since ROIs define 3D positions in 3-space. It is desired to collect images around the ROI. A projection of the 3D position to 2D is performed. The camera pose is used to perform the projection / collection. The point cloud is used for occlusion detection as shown in FIG. 11.

[0161] Further steps include, at 515, downsampling the 3D point cloud representation to obtain a set of candidate regions of interest 525-1 to 525-7; filtering the candidate regions of interest to select those of the candidate regions of interest having appearance changes characterizing different visual states as selected regions of interest 535 based at least in part on the camera pose; generating region of interest images 415 for the selected regions of interest from corresponding ones of the video frames based at least in part on the camera pose; and training a deep learning recognition model based on the region of interest images at step 419.

[0162] Some examples further include, as 405, using the trained deep learning model to determine a visual state of the object with a digital camera coupled to the processor.

[0163] Some embodiments further include, as step 1013, providing augmented reality instructions related to the object to the user based on the determined state. For example, a database can be included in the AR system having instructions to superimpose on a view of the object stored in association with the visual state of the object. Such a database can reside in local storage in the device 569, or can be stored in the cloud as described in connection with FIGS. 1 and 2, or both. When a state is determined, the database is queried for corresponding instructions, and they are superimposed on the view of the object and displayed on the screen of the device 509. The display 24 generally represents such a screen.

[0164] Some examples further include, as step 404, repositioning the digital camera relative to the object before determining the state of the object.

[0165] In some cases, downsizing the three-dimensional point cloud representation to obtain candidate regions of interest includes removing outliers, as shown at 511.

[0166] In some cases, filtering the candidate regions of interest to select those candidate regions of interest that have an appearance change includes acquiring and zooming images at 537.

[0167] In some examples, determining the state of the object includes capturing 709 images of those regions of interest that have occupied voxels of interest; and providing corresponding zoomed local images and view poses 711 for each of the regions of interest that have occupied voxels to the recognition model 545.

[0168] In some embodiments, the model includes a multi-stream convolutional neural network long short-term memory model; and determining the state of the object includes separating the region of interest into multiple branches 803-1 ... 803-K, as in 805, extracting features from corresponding zoomed local images using a multi-stream convolutional neural network long short-term memory model, as in 809, aggregating spatial information by concatenating the extracted features; and aggregating temporal information by pooling spatial information for multiple temporal steps, as in 813, and providing the pooled information to a long short-term memory layer 815 to obtain a fully connected layer 817 for classification to determine the state of the object.

[0169] One or more embodiments further include reducing susceptibility to at least one of occlusion and camera instability by not capturing images for regions of interest that do not have occupied voxels of interest, as shown at 708, 709.

[0170] In one or more embodiments, the object 502 includes an information technology asset to be repaired, a digital camera, and a processor that is part of the mobile device 569, 54A, and providing the augmented reality instructions includes illustrating a repair procedure for the information technology asset. A further step includes repairing the asset according to the instructions.

[0171] One or more embodiments of the present invention, or elements thereof, may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and operative to perform the method steps of the example. FIG. 22 illustrates a computer system that may be useful for implementing one or more aspects or elements of the present invention, or both, and is representative of a cloud computing node according to an embodiment of the present invention. Referring now to FIG. 22, the cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to limit in any way the scope of use or functionality of the embodiments of the present invention described herein. Instead, the cloud computing node 10 may implement and / or perform any of the functions described herein above.

[0172] Cloud computing node 10 includes computer system / server 12, which may operate with numerous other general purpose or special purpose computing system environments or configurations. Known computing systems, environments or configurations, or combinations thereof, suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments, including any of the above systems or devices.

[0173] The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing system devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0174] 22, computer system / server 12 in cloud computing node 10 is shown in the form of a general purpose computing device. Components of computer system / server 12 include, but are not limited to, one or more processors, or processing units 16, system memory 28, and a bus 18 connecting various system components, including system memory 28, to processor 16.

[0175] Bus 18 may represent any of several types of one or more bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerator port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro-Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Interconnect (PCI) bus.

[0176] The computer system / server 12 typically includes a variety of computer readable media, which may be any available media that is accessible by the computer system / server 12, including both volatile and non-volatile media, removable and non-removable media.

[0177] The system memory 28 may include computer readable media such as volatile memory, such as random access memory (RAM) 30, or cache memory 32, or both. The computer server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown and typically referred to as a "hard drive"). Although not shown, a removable, non-volatile magnetic disk drive (e.g., a "floppy disk") and an optical disk drive may be provided for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such an example, each may be connected to the bus 18 by one or more media interfaces. As further shown and described below, the memory 28 may include at least one program product having a set (at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0178] The programs / utilities 40 include a set (at least one) of program modules 42, which by way of example and not limitation, may include an operating system, as well as one or more application programs, other program modules, and program data stored in memory 28. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methodologies of embodiments of the present invention as described herein.

[0179] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24; one or more devices that may allow a user to interact with the computer system / server 12, or any device that allows the computer system / server 12 to communicate with one or more computing devices (e.g., network cards, modems, etc.), or both. Such communication may occur through an input / output (I / O) interface 22. Additionally, the computer server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (the Internet), or combinations thereof, through a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 through a bus 18. It should be understood that other hardware and / or software components, not shown, could be used in combination with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, and external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0180] Thus, one or more embodiments may employ software operating on a general purpose computer or workstation. With reference to FIG. 22, such an implementation may include, for example, a processor 16, memory 28, and an input / output interface 22 to a display 24, and external device(s) 14, such as a keyboard, pointing device, etc. As used herein, the term "processor" is intended to include any processing device, such as, for example, a CPU (Central Processing Unit), or other form of processing circuitry, or both. Furthermore, the term "processor" may refer to one or more individual processors. Furthermore, the term "processor" may refer to one or more individual processors. The term "memory" is intended to include memory associated with a processor or CPU, and is intended to include, for example, RAM (Random Access Memory) 30, ROM (Read Only Memory), fixed memory devices (e.g., hard drive 34), removable memory devices (e.g., diskettes), flash memory, etc. Additionally, the phrase "input / output interface" as used herein is intended to contemplate, for example, an interface to one or more mechanisms for inputting data into the processing unit (e.g., a mouse) and one or more subsequent mechanisms for providing related results to the processing unit (e.g., a printer). The processor 16, memory 28, and input / output interface 22 may be interconnected, for example, via a bus 18 extending as part of the data processing unit 12. For example, suitable interconnections via the bus 18 may be provided to a network interface 20, such as a network card, which may provide for interfacing to a computer network, and to a media interface, such as a diskette or CD-ROM drive, which may provide for interfacing to a suitable media.

[0181] Thus, computer software containing instructions or code for carrying out the methodologies of the present invention, as described herein, can be stored in one or more associated memory devices (e.g., ROM, fixed or removable memory) and can be loaded partially or entirely into a CPU (e.g., in RAM) and implemented when ready for use. Such software can include, but is not limited to, firmware, persistent software, microcode, etc.

[0182] A data processing system for storing and / or executing program code will include at least one processor 16 coupled directly or indirectly to memory elements 28 through a system bus 18. The memory elements may include local memory used during the actual implementation of the program code, bulk storage, and a cache memory 32 providing temporary storage of at least some of the program code to reduce the number of times the code must be recalled from bulk storage during implementation.

[0183] Input / output, or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I / O controllers.

[0184] Network adapters 20 may also be coupled to the system to enable the data processing system to become coupled to other data processing systems, or remote printers, or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are some of the types of network adapters currently available.

[0185] As used herein, including the claims, a "server" includes a physical data processing system that runs a server program (e.g., system 12 in FIG. 22). Such a physical server may or may not include a display and keyboard.

[0186] One or more embodiments may be implemented, at least in part, in a cloud or virtual machine environment, by way of example and not limitation. See back to Figures 1-2 and the associated text. For example, it is envisioned that a cloud-based service 96 for fine-grained level visual recognition in mobile augmented reality would be located within layer 90.

[0187] Any of the methods described herein may include the additional step of providing a system including distinct software modules embodied on a computer-readable storage medium; the modules may include any or all of the appropriate elements, for example, as set forth in the block diagrams and / or described herein; by way of example and not limitation, any one, some, or all of the described module blocks, or sub-modules, or sub-blocks, or both, are described. The method steps may then be performed as described above using the distinct software modules, or sub-modules, or both, of the system executing on one or more hardware processors 16. Additionally, a computer program product may include a system having distinct software modules, including a computer-readable storage medium having code adapted to be implemented to perform one or more method steps described herein.

[0188] One example user interface that may be used in some cases is HyperText Markup Language (HTML) code provided by a server or the like to a browser on a user's computing device, which is parsed by the browser on the user's computing device to generate a Graphical User Interface (GUI).

[0189] Manufacturing Details of Exemplary Systems and Products

[0190] The present disclosure may be a system, method, or computer program product, or a combination thereof, including a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform the features of the present disclosure.

[0191] A computer readable recording medium may be a tangible device capable of holding and storing a plurality of instructions for use by an instruction execution device. A computer readable medium may be, for example, but not limited to, an electrical recording device, a magnetic recording device, an optical recording device, an electro-magnetic recording device, a semiconductor recording device, or any suitable combination thereof. More specific examples of computer readable recording media include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card, or a mechanically encoded device having protruding structures within a groove having instructions recorded therein, and any suitable combination thereof. As used herein, a computer-readable recording medium is not to be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves such as wave guides or other communications media (e.g., light pulses passing through fiber optic cable), or electrical signals communicated through wires.

[0192] The computer programs described herein can be downloaded from a computer readable recording medium into each computing / processing device, or can be downloaded to an external computer or external recording device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, and combinations thereof. The network can include copper cables, fiber optics, wireless networks, routers, firewalls, switches, gateway computers, and edge servers, and combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions into a computer readable recording medium in each computing / processing device for storage.

[0193] The computer readable program instructions for carrying out the operations of the present invention can be either source code or object code written in any combination of programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or one or more procedural programming languages, such as Smalltalk®, object oriented programming languages ​​such as C++, the “C” programming language, or similar programming languages. The computer readable program instructions can be executed entirely on the user computer, partially on the user computer as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user computer through any type of network, including a local area network (LAN), a wide area network (WAN), or the connection can be to an external computer (e.g., through an Internet service provider). In some embodiments, computer readable program instructions may be executed by electrical circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), and state information of the computer readable program instructions may be used to personalize the electrical circuitry to perform features of the present invention.

[0194] Features of the invention described herein have been described with reference to flowchart instructions and / or block diagrams of methods, apparatus (systems), and computer-readable recording media and computer program products according to embodiments of the invention. It will be understood that any combination of flowchart illustrations and / or block diagrams and / or blocks in flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0195] The computer-readable program instructions can be provided to a general purpose computer, a special purpose computer, or other processor or other programmable data processing device to produce a machine, which when executed by the processor of the computer or other programmable data processing device produces means for implementing the functions / operations specified in the block or blocks of the flowcharts and block diagrams, or a combination thereof. These computer-readable program instructions that direct a computer, programmable data processing device, and other device, or a combination thereof, to function in a particular manner can also be stored on a computer-readable recording medium, and the computer-readable recording medium having instructions stored therein constitutes an article of manufacture containing instructions that implement the functional / operational features specified in the block or blocks of the flowcharts and block diagrams, or a combination thereof.

[0196] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device and cause a computer implemented process to perform a series of operational steps on the computer, other programmable apparatus, or other device to implement the functions / acts specified in the blocks of the flowcharts and block diagrams or in a combination thereof.

[0197] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and possible implementation operations of systems, methods, and computer programs according to various embodiments of the present invention. In this respect, the flowcharts or block diagrams may represent modules, segments, or portions of instructions, which include one or more executable instructions for implementing a particular logical function(s) (or functions). In some alternative implementations, the functions described in the blocks may be performed other than as illustrated. For example, two blocks shown in succession may actually be executed substantially simultaneously, partially or fully overlapped in time, depending on the functionality involved, or the blocks may sometimes be executed in reverse order. It is also noted that the illustration of the blocks in the block diagrams and / or the illustration of the flowcharts or combinations thereof may be implemented by a system based on special purpose hardware that performs a particular function or operation or executes specific purpose hardware and computer instructions.

[0198] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limiting to the disclosed embodiments. Many modifications or variations will be obvious to those skilled in the art without departing from the scope and spirit of the present disclosure. The terms used in this specification are selected to best explain the principles of the present embodiments, practical applications, or technical improvements over the art found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. Gaining access to a three-dimensional point cloud representation of the object, including the pose of the scanning digital camera and the corresponding video frames; downsampling the three-dimensional point cloud representation to obtain a set of candidate regions of interest; filtering the candidate regions of interest to select as selected regions of interest those candidate regions of interest having appearance changes that characterize different visual states based at least in part on the pose of the digital camera; generating a region of interest image for the selected region of interest from a corresponding one of the video frames; and Training a deep learning recognition model based on the image of the region of interest. The method includes:

2. and determining a visual state of the object with the digital camera coupled to a processor using the trained deep learning recognition model. The method of claim 1 , comprising:

3. and providing an augmented reality indication to a user related to the object based on the determined state. The method of claim 2 , comprising:

4. and repositioning the digital camera relative to the object before determining the state of the object. The method of claim 3 , comprising:

5. The downsampling of the three-dimensional point cloud representation to obtain a set of candidate regions of interest includes removing outliers. The method of claim 4 , comprising:

6. Filtering the candidate regions of interest to select the candidate regions of interest having a change in appearance may include collecting and zooming images of the selected regions of interest. The method of claim 5 , comprising:

7. Determining the state of the object includes: capturing an image of the region of interest in which voxels of interest are occupied; providing the deep learning recognition model with corresponding zoomed local images and view poses for each of the regions of interest in which the voxels of interest are occupied; The method according to any one of claims 4 to 6, comprising:

8. The deep learning recognition model includes a multi-stream convolutional neural network long short-term memory model; and Determining the state of the object includes: Separating the region of interest into a plurality of branches; extracting features from the corresponding zoomed local images using the multi-stream convolutional neural network long short-term memory model convolutional neural network; aggregating spatial information by concatenating the extracted features; aggregating the temporal information by pooling the spatial information for multiple temporal steps; and providing the pooled information to a long short-term memory layer to obtain a fully connected layer for classification to determine the state of the object; The method of claim 7, comprising:

9. Further, to reduce sensitivity to at least one of occlusion and camera instability, avoiding capturing images for the region of interest that does not have occupied voxels of interest.

9. The method of claim 7 or 8, comprising:

10. The object includes an information technology asset to be repaired, the digital camera and processor are part of a mobile device, and the providing of the augmented reality instructions indicates a repair procedure for the information technology asset. The method according to any one of claims 3 to 9, comprising:

11. Memory, and at least one processor coupled to the memory, the at least one processor operative to: Gaining access to a three-dimensional point cloud representation of the object, including the pose of the scanning digital camera and the corresponding video frames; downsampling the three-dimensional point cloud representation to obtain a set of candidate regions of interest; filtering the candidate regions of interest to select as selected regions of interest those candidate regions of interest having appearance changes that characterize different visual states based at least in part on the pose of the digital camera; generating a region of interest image for the selected region of interest from a corresponding one of the video frames; and training a deep learning recognition model based on the image of the region of interest; Device.

12. the at least one processor is further operative to determine a visual state of the object with the digital camera coupled to the processor using the trained deep learning recognition model.

12. The apparatus of claim 11.

13. the at least one processor is further operative to provide an augmented reality indication related to the object to a user based on the determined state.

13. The apparatus of claim 12.

14. the at least one processor is further operative to reposition the digital camera relative to the object before determining the state of the object.

14. The apparatus of claim 13.

15. The downsampling of the three-dimensional point cloud representation to obtain a set of candidate regions of interest includes removing outliers. The filtering of the candidate regions of interest to select the candidate regions of interest having a change in appearance may include collecting and zooming images of the selected regions of interest. The apparatus of claim 14 , comprising:

16. capturing an image of the region of interest in which voxels of interest are occupied; providing the deep learning recognition model with corresponding zoomed local images and view poses for each of the regions of interest in which the voxels of interest are occupied; 16. The apparatus of claim 14 or 15, comprising:

17. The deep learning recognition model includes a multi-stream convolutional neural network long short-term memory model; and Determining the state of the object includes: Separating the region of interest into a plurality of branches; extracting features from the corresponding zoomed local images using the multi-stream convolutional neural network long short-term memory model convolutional neural network; aggregating spatial information by concatenating the extracted features; aggregating the temporal information by pooling the spatial information for multiple temporal steps; and providing the pooled information to a long short-term memory layer to obtain a fully connected layer for classification to determine the state of the object; and the at least one processor is further operative to avoid capturing images for the region of interest that does not have occupied voxels of interest to reduce sensitivity to at least one of occlusion and camera instability; 17. The apparatus of claim 16.

18. The object includes an information technology asset to be repaired, the digital camera and processor are part of a mobile device, and the providing of the augmented reality instructions indicates a repair procedure for the information technology asset. The apparatus of claim 13 , comprising:

19. A computer program for causing a computer to carry out the method according to any one of claims 1 to 10.

Images