Object detection and anchor position and orientation resolution for augmented reality applications

Abstract

Identifying a feature of interest of an object within a physical environment from a video of the physical environment. Obtaining a model of the object, the model including dimensions of the object. Maintaining a first array with pixel coordinates of the identified feature within the video. Maintaining a second array with the dimensions of the feature obtained from the model. Providing a processing framework for using the first array and the second array and estimating, updating and tracking anchor poses in the physical environment within the video and any AR generated objects placed in the video. Providing the framework to an augmented reality (AR) app to initiate an AR session and maintain the session.

Description

Technical Field

[0001] This application relates to augmented reality (AR) sessions, and more specifically to augmented reality (AR) applications. Background Technology

[0002] For the purpose of configuring applications for AR sessions, augmented reality (AR) applications rely on anchor point objects identified in real-world three-dimensional (3D) space. The application uses the anchor point's known position and orientation in 3D space to resolve the position and orientation of other real-world objects captured in the video feed relative to the anchor point. This allows the application to map the physical environment within the video feed to its objects during a session.

[0003] Once anchor points are used to map the physical environment, AR-generated objects can be rendered during a session and overlaid within the video feed. Traditionally, anchor points are identified either by markers placed on real-world objects within the physical environment or by point cloud 3D models. Marking adds to the costs associated with manufacturing markers, requiring the physical environment to be marked, finding suitable surfaces within the physical environment to place the markers, and the markers may degrade in quality over time due to normal wear and tear or conditions within the physical environment. Point cloud models require additional user interaction to align physical objects from the physical environment with their representations in the model, causing session initiation delays. Furthermore, a single model cannot adequately represent the different configurations of complex physical objects within the physical environment (e.g., open doors, missing modules in racks, etc.).

[0004] Anchoring methods based on pre-established tags and session-initiated mapping from anchors to models are costly, time-consuming, and flawed. Therefore, AR applications require improved real-world object detection with anchor resolution during AR session initiation, mapping the physical environment to the video feed. Summary of the Invention

[0005] In various embodiments, systems and methods for anchor object detection and anchor position and orientation (hereinafter “pose”) resolution during augmented reality (AR) session initiation are presented. During AR session initiation, features of objects are identified from a video feed of the physical environment. These features are mapped to screen coordinates within a 2D array. A computer-aided design (CAD) model of the object is obtained, and the CAD dimensions and orientation of the features are added to a 3D array. A perspective n-point (PnP) algorithm is processed using the internal features of the camera providing the video feed, the 2D array, and the 3D array to estimate the pose of the anchor object in the real world, as well as what is depicted in the video feed at any given point in time. This provides an AR framework to the AR app during AR session initialization; the framework is maintained for the AR app to dynamically track and update the anchor pose as the video feed changes and as AR-generated objects overlap within the video feed via the AR application.

[0006] According to a first aspect, this application provides a method comprising the steps (1)-(4). In step (1), during the initiation of an AR session of an augmented reality (AR) application (app) within a physical environment containing a real-world object, features associated with the real-world object are identified in a video of the AR app. In step (2), pixel coordinates of the features within the video are maintained. In step (3), the size and dimensions of the features are maintained based on a model including the size and dimensions of the real-world object. In step (4), the position and orientation of the real-world object as depicted in the video are estimated based on the pixel coordinates, the size, and the dimensions.

[0007] According to the first aspect above, the method further includes processing the method into a framework for the AR app to track and update the position and orientation of the real-world object within the video as an anchor object for the AR session.

[0008] According to the first aspect above, the method includes processing the frame to track and update the position and orientation associated with an AR-generated object during the AR session, the AR-generated object overlapping within the video by the position and orientation of the AR app relative to the anchor object.

[0009] According to the first aspect above, the method also includes providing the framework to the AR app via an application programming interface.

[0010] According to the first aspect above, the identifier further includes identifying the features from the video based on known features associated with the real-world object.

[0011] According to the first aspect above, the identifier further includes selecting the feature from a plurality of known features associated with the real-world object.

[0012] According to the first aspect above, maintaining the pixel coordinates further includes maintaining the pixel coordinates of the features of the real-world object and the additional pixel coordinates of additional features in a two-dimensional (2D) array.

[0013] According to the first aspect above, maintaining the size further includes maintaining the size and dimensions of the features of the real-world object and the additional size and dimensions of the additional features in a three-dimensional (3D) array.

[0014] According to the first aspect above, the estimation further includes obtaining internal camera parameters from the AR app for the camera used to supply the video.

[0015] According to the first aspect above, the estimation further includes using the 2D array, the 3D array, and the internal camera parameters to process a perspective n-point algorithm to estimate the position and orientation of the real-world object within the video.

[0016] According to a second aspect, this application provides a method comprising performing the following steps (1)-(6). In step (1), a computer-aided design (CAD) model of a terminal is maintained. In step (2), an AR session initiation request is received from an augmented reality (AR) application (app), the AR app including a video of a physical environment, wherein the terminal is located within the physical environment. In step (3), three or more features of the terminal are identified from the video. In step (4), the size and dimensions of the three or more features are obtained from the CAD model. In step (5), internal camera parameters of a camera supplying the video are obtained. In step (6), the position and orientation of the terminal within the video are estimated using the pixel coordinates of the three or more features presented on a screen on a device including the camera, the size, the dimensions, and the internal camera parameters, and an AR session of the AR app is initialized in response to the AR session initiation request.

[0017] According to the second aspect above, the method further includes continuously repeating the use during the AR session to track and update the location and orientation of the terminal within the video, and tracking and updating the location and orientation of AR-generated objects within the video during the AR session.

[0018] According to the second aspect above, receiving further includes identifying the AR session initiation request as a maintenance or service action to be performed by an engineer at the terminal.

[0019] According to the second aspect above, the identifier further includes selecting the three or more features from a plurality of features based on visual attributes of the three or more features detected in the video.

[0020] According to the second aspect above, obtaining the size further includes maintaining the pixel coordinates, the size, and the dimensions of the three or more features in a two-dimensional (2D) array and a three-dimensional (3D) array.

[0021] According to the second aspect above, a perspective n-point (PnP) algorithm is further used, which involves processing the 2D array, the 3D array, and the internal camera parameters, to estimate the position and orientation of the terminal within the video.

[0022] According to the second aspect above, the method further includes tracking the video after the AR session is initiated, and updating the location and orientation of the terminal within the video and any AR-generated objects presented within the video during the AR session.

[0023] According to the second aspect above, the method further includes providing the method as a cloud-based Software as a Service (SaaS) to the AR app.

[0024] According to a third aspect, this application provides a system comprising a cloud server. The cloud server includes at least one processor and a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium includes executable instructions. When provided by the non-transitory computer-readable storage medium to and executed by the at least one processor, the executable instructions cause the at least one processor to perform operations including the following steps (1)-(3). In step (1), a computer-aided design (CAD) model of a terminal is maintained. In step (2), an application programming interface (API) is provided to an augmented reality (AR) application (app). In step (3), the AR app is interacted with via the API through methods 1)-5) to provide AR session initialization and AR session tracking based on features of the terminal located in the physical environment detected in the video during an AR session. In method 1), the pixel coordinates of the features of the terminal are mapped to their positions on the screen of the device presenting the video. In method 2), the size and dimensions and pixel coordinates of the features obtained from the CAD model are maintained. In method 3), internal camera parameters of the camera supplying the video are obtained. In mode 4), the pixel coordinates, the size, the dimensions, and the internal camera parameters are used to process the perspective n-point (PnP) algorithm to estimate the position and orientation of the terminal within the screen containing the video for AR session initialization. In mode 5), during the AR session representing the AR application, the position and orientation of the terminal and any AR-generated objects are updated.

[0025] According to the third aspect above, the terminal is a point-of-sale (POS) terminal, a self-service terminal (SST), an ATM, or a self-service machine.

[0026] According to a fourth aspect, this application provides a method that performs the following steps (1)-(4). In step (1), during the initiation of an AR session of an augmented reality (AR) application (app) within a physical environment containing a real-world object, features associated with the real-world object are identified in a video of the AR app. In step (2), the pixel coordinates of the features within the video are maintained. In step (3), the size and dimensions of the features are maintained based on a model including the size and dimensions of the real-world object. In step (4), the position and orientation of the real-world object as depicted in the video are estimated based on the pixel coordinates, the size, and the dimensions.

[0027] According to the fourth aspect above, the method performs any one of the following steps (1)-(3). In step (1), the method is processed into a frame for the AR app to track and update the position and orientation of the real-world object within the video as an anchor object for the AR session. In step (2), in addition to step (1) above, the method further includes the step of processing the frame to track and update the position and orientation associated with an AR-generated object during the AR session, the AR-generated object overlapping the position and orientation of the AR app relative to the anchor object within the video. In step (3), in addition to step (2) above, the method further includes the step of providing the frame to the AR app via an application programming interface.

[0028] According to the fourth aspect above, the method performs any one of the steps (1)-(2). In step (1), the identification further includes identifying the feature from the video based on known features associated with the real-world object. In step (2), the identification further includes selecting the feature from a plurality of known features associated with the real-world object.

[0029] According to the fourth aspect above, maintaining the pixel coordinates further includes maintaining the pixel coordinates of the features of the real-world object and the additional pixel coordinates of additional features in a two-dimensional (2D) array.

[0030] According to the fourth aspect above, the method performs any one of the steps (1)-(3). In step (1), maintaining the size further includes maintaining the size and dimensions of the features of the real-world object and the additional size and dimensions of the additional features in a three-dimensional (3D) array. In step (2), in addition to including step (1) above, the method further includes the step of estimation, which further includes obtaining internal camera parameters from the AR app for the camera used to supply the video. In step (3), in addition to including step (2) above, the method further includes the step of estimation, which further includes using the 2D array, the 3D array, and the internal camera parameters to process a perspective n-point algorithm to estimate the position and orientation of the real-world object within the video.

[0031] According to a fifth aspect, this application provides a method comprising performing the following steps (1)-(6). In step (1), a computer-aided design (CAD) model of a terminal is maintained. In step (2), an AR session initiation request is received from an augmented reality (AR) application (app), the AR app including a video of a physical environment, wherein the terminal is located within the physical environment. In step (3), three or more features of the terminal are identified from the video. In step (4), the size and dimensions of the three or more features are obtained from the CAD model. In step (5), internal camera parameters of a camera supplying the video are obtained. In step (6), the position and orientation of the terminal within the video are estimated using the pixel coordinates of the three or more features within a screen presented on a device including the camera, the size, the dimensions, and the internal camera parameters, and an AR session of the AR app is initialized in response to the AR session initiation request.

[0032] According to the fifth aspect above, the method includes performing any one of the steps (1)-(4). In step (1), the use to track and update the location and orientation of the terminal within the video is repeated continuously during the AR session, and the location and orientation of AR-generated objects within the video are tracked and updated during the AR session. In step (2), receiving further includes identifying the AR session initiation request as a maintenance or service action to be performed by an engineer at the terminal. In step (3), the identification further includes selecting the three or more features from a plurality of features based on visual attributes of the three or more features detected in the video. In step (4), the method is provided to the AR app as a cloud-based Software as a Service (SaaS).

[0033] According to the fifth aspect above, the method includes performing any one of the steps (1)-(3). In step (1), obtaining the size further includes maintaining the pixel coordinates, the size, and the dimensions of the three or more features in a two-dimensional (2D) array and a three-dimensional (3D) array. In step (2), in addition to step (1) above, the method includes using a perspective n-point (PnP) algorithm that further includes using the 2D array, the 3D array, and the internal camera parameters to estimate the position and orientation of the terminal within the video. In step (3), in addition to step (2) above, the method includes tracking the video after the AR session is initiated and updating the position and orientation of the terminal within the video and any AR-generated objects presented within the video during the AR session. Attached Figure Description

[0034] Figure 1 This is a diagram of a system for anchor object detection and anchor pose resolution during AR session initiation, according to an example embodiment.

[0035] Figure 2 This is a flowchart of a method for anchor object detection and anchor pose resolution during an AR session, according to an example embodiment.

[0036] Figure 3 This is a flowchart of another method for anchor object detection and anchor pose resolution during an AR session, according to an example embodiment. Detailed Implementation

[0037] As stated above, AR session initiation is problematic for AR applications, where the physical objects of the tag-based anchoring method are tagged with barcodes or Quick Response (AR) encoding, and the physical objects of the model-based anchoring method must be mapped to a potentially incomplete model. Furthermore, both methods can experience the same session initiation latency experienced by the end user of the AR application (hereinafter referred to as "app").

[0038] The teachings provided herein offer a dynamic and real-time technique for identifying anchor objects and / or selecting anchor objects from images or image frames in a video feed (hereinafter referred to as "video feed"). The video feed is captured from the physical environment by a camera of a user-operated device during the initiation of an AR session in an AR app. The attention-grabbing features of the anchor object are detected by matching known features of the anchor object with features identified from the video feed of the physical environment. Screen coordinates of the features used in the video are fed and labeled within a two-dimensional (2D) array. The 2D array maintains a mapping of the attention-grabbing feature of the anchor to the corresponding screen position on the display for each of the attention-grabbing features; for example, feature z has x-pixel (width) coordinates and y-pixel (height) coordinates within the video feed. A computer-aided design (CAD) model associated with the object is obtained. The CAD model is used to derive a 3D array or matrix. The 3D array contains the dimensions and initial orientation associated with the attention-grabbing features of the object relative to the starting point of the CAD model. Detected features are recorded in the 2D array with their labels and screen coordinates. Next, the internal camera parameters of the camera providing the video feed are obtained for factors such as the camera's focal length and lens distortion. The pose (orientation and position) of the anchor point relative to the camera within the physical environment is estimated using a 2D array of features, a 3D array of features, the internal camera parameters, and a perspective n-point (PnP) algorithm. This provides an initial and ongoing framework by which the anchor point pose, as well as any AR-generated objects overlapping within the AR session, are continuously recalculated and updated as the video feed changes during the AR app session.

[0039] As discussed in this article and below, for the purpose of initializing an AR session for an AR app in a given physical environment, anchor objects do not require tagged physical environment labels and point cloud model mappings. This allows the AR framework for the physical environment to be initialized much faster than traditionally available AR frameworks for AR apps, because the framework does not require detection of planes, image labels, or 3D point cloud models.

[0040] Figure 1 This is a diagram of a system 100 for anchor object pose resolution during an AR session, according to an example embodiment. It should be noted that the components are shown schematically in a greatly simplified form, with only those components relevant to understanding the embodiment depicted.

[0041] Furthermore, for illustrative purposes only, the following are shown ( Figure 1 The various components (identified in the text) are presented, along with their arrangement. It should be noted that other arrangements with more or fewer components are possible without departing from the teachings on anchor object pose resolution during AR sessions presented below.

[0042] System 100 includes a cloud 110 or server 110 (hereinafter referred to as "cloud 110"), an AR provider server 120, and an AR operating device 130. Cloud 110 includes a processor 111 and a non-transitory computer-readable storage medium 112 (hereinafter referred to as "medium 112") containing executable instructions for an AR anchor resolver and tracker 113. Processor 111 receives or is provided with executable instructions from medium 112, causing processor 111 to perform the operations discussed herein and hereinafter with respect to 113.

[0043] AR provider server 120 includes at least one processor 121 and medium 122, the medium containing instructions for ARapp interface 123 and model manager 124. Processor 121 receives or is provided with instructions from medium 122, causing processor 121 to perform the operations discussed herein and hereinafter with respect to 123.

[0044] Each AR operating device 130 includes a processor 131, a camera 132, a display 133, and a medium 134 containing executable instructions for an AR app 135 and an AR app application programming interface (API) 136. The processor 131 receives or is provided with executable instructions from the medium 134, causing the processor 131 to perform the operations discussed herein and hereinafter with respect to 135 and 136.

[0045] System 100 is described from the perspective of a complex real-world object placed within a physical environment. It should be noted that this is done for illustrative and understanding purposes and may not necessarily be the case in different embodiments of the teaching content. A complex real-world object is a transaction terminal, as exemplified below. The terminal may be an automated teller machine (ATM), a self-service terminal (SST), a point-of-sale (POS) terminal, or a self-service machine. However, it should be noted that other complex objects, such as vehicles, robots, mechanical parts, equipment, etc., may also be used. In this embodiment, a complex real-world object is an entire room that includes other objects known to be present in the room.

[0046] A complex real-world object is associated with a model data structure, which defines the components of the complex real-world object by the 3D dimensions (height, width, and length) of each component and the various available orientations of each component relative to each other within the complex real-world object. In this embodiment, the model data structure is a CAD model of the complex real-world object; however, it should be noted that other customer-defined model data structures may also be used.

[0047] The AR provider offers AR app 135 to the user of device 130 for the purpose of providing AR services to the user through AR app interface 123. For example, the AR provider may provide AR app interface 123 and the corresponding AR app 135 to customer engineers performing maintenance and support on an ATM (e.g., a type of composite real-world object). The model data structure of the ATM is maintained by model manager 124 on AR provider service 120.

[0048] The teachings presented herein and below enhance AR app 134, provided by the AR provider, to interact with AR app API 136. AR app API 136 interacts with model manager 124 and AR anchor resolver and tracker 113 for the purpose of establishing an AR framework to initialize an AR session via AR app 135, thereby establishing the pose of anchor objects, and for the purpose of tracking and updating the pose of anchor objects and any AR-generated objects during the AR session between AR app interface 123 and AR app 135.

[0049] During the initialization of the AR session, AR app 135 uses AR app API 136 to obtain a 3D coordinate array (derived from a CAD model) of detected features of complex real-world objects presented in the user's physical environment. When the user operates device 130 and initiates an AR session between AR app 135 and AR app interface 123, the array, along with real-time video captured by camera 132 of the physical environment, is provided to the AR anchor resolver and tracker.

[0050] For the purpose of identifying one or more attention-grabbing features of complex real-world objects within a physical environment, the AR anchor resolver and tracker 113 perform computer vision processing on images from a video depicting the physical environment. The one or more attention-grabbing features are predefined for complex real-world objects and maintained by the AR anchor resolver and tracker 113 based on their relatively small physical size and unique visual characteristics. The AR anchor resolver and tracker 113 identifies three or more features from initial image frames provided with the video. The pixel coordinates of the features within the video are maintained in a 2D array. The 3D dimensions of the features are maintained in a 3D array using a model data structure. The internal camera parameters of camera 132 are obtained from AR app 135 via AR app API 136. By way of example only, the internal camera parameters include the camera's focal length, lens distortion, etc.

[0051] Next, the AR anchor parser and tracker 113 establish a framework through which complex real-world objects are used to define anchor objects for the AR session, and through which the poses of anchor objects appearing in the real world can be estimated, tracked, and updated within the video during the session. No AR-generated objects need to be tracked, as they are attached to the AR anchors. The framework uses a 2D array of detected features of the objects, a 3D array of object features, and internal camera parameters to process the PnP algorithm. Therefore, as the video changes, the AR anchor parser and tracker 113 use the framework to update the poses of the anchor objects within the video and thus update the attached AR-generated objects. The video is streamed directly to the user's display 133 during the session.

[0052] In this embodiment, the complex real-world object is an ATM, and the user is a customer engineer performing maintenance on the ATM. The AR provider is an ATM support organization that provides an AR maintenance app 135 to customers via an AR device 130, which can be a mobile phone, tablet, laptop, or wearable processing device such as headphones or goggles. When app 135 initializes the ATM, the starting point and pose within the video captured by camera 132 are estimated using the framework discussed above. AR app 135 interacts with AR app interface 123 to identify the component model of the ATM that the engineer needs to remove and replace by using the starting point and pose of the ATM to identify the module, and plays an instructional video on top of the live video captured by camera 132, and displays the live video and instructional video superimposed on display 133 for the engineer to view. When the engineer inserts his hand to grasp the module if it is not the correct module, interface 123 can use the framework provided by AR anchor resolver and tracker 113 to place the AR-rendered X on the hand within the video. The session continued until the engineer successfully replaced the module using app135, interface 123, and based on the framework provided by the AR anchor parser and tracker 113.

[0053] In one embodiment, the AR anchor resolver and tracker 113 are processed on the AR provider server 120. In another embodiment, the model manager 125 is processed on the cloud 110. In yet another embodiment, the AR app 135 is further enhanced to process the AR anchor resolver and tracker 113. In yet another embodiment, the AR anchor resolver and tracker 113 are provided as Software as a Service (SaaS) to the AR app 135 and / or the AR app interface 123.

[0054] System 100 improves the initialization time of AR app 135 and the response time during AR sessions with interface 123 by providing a framework that is neither based on marked surfaces within the physical environment nor on mapping real-world objects to a point cloud model. Instead, object features are used to dynamically resolve anchor points for the AR session using known dimensions of features from a model data structure (e.g., a CAD model). The origin and pose of the anchor object are estimated, and the framework is constructed from a 2D array of features, a 3D array of features, and internal camera features of camera 132. The framework is processed by a PnP algorithm to continuously maintain, track, and update the pose of the anchor object and the pose of any AR-generated objects relative to the anchor object. This improves the response time of AR app 135 and AR interface 123.

[0055] Now for reference Figure 2 and Figure 3 The above-mentioned embodiments and other embodiments are discussed. Figure 2 This is a flowchart of a method 200 for anchor object pose resolution during an AR session according to an example embodiment. The software module implementing method 200 is referred to as an "AR anchor pose manager". The AR anchor pose manager is implemented as a programmable executable instruction residing in memory and / or a non-transitory computer-readable (processor-readable) storage medium, and executed by one or more processors of one or more devices. The processor of the device executing the AR anchor pose manager is specifically configured and programmed to process the AR anchor pose manager. The AR anchor pose manager may access one or more network connections during its processing. The connection may be wired, wireless, or a combination of wired and wireless.

[0056] In one embodiment, the device executing the AR anchor pose manager is cloud 110. In another embodiment, the device executing the AR anchor pose manager is server 110. In another embodiment, the device executing the terminal media baseline predictor is AR provider server 120. In yet another embodiment, the AR anchor pose manager is all or a combination of 113, 124, and / or 136.

[0057] At 210, the AR anchor pose manager identifies features associated with real-world objects in the video of AR app 135 during the initiation of an AR session within the physical environment. The video depicts the real-world app within the physical environment.

[0058] In one embodiment, at 211, the AR anchor pose manager identifies features based on known features associated with real-world objects. In another embodiment, at 212, the AR anchor pose manager selects features from a plurality of known features associated with real-world objects.

[0059] At 220, the AR anchor point pose manager maintains the pixel coordinates of features within the video. These pixel coordinates are identified within the screen displayed on the device's display 133, where the features are located within the screen. The x and y coordinates are in 2D.

[0060] In one embodiment, at point 221, the AR anchor pose manager maintains the pixel coordinates of features of the real-world object and the additional pixel coordinates of additional features in a 2D array. In this embodiment, the additional pixel coordinates are used for one or more additional features, and the additional pixel coordinates are maintained in a 2D array for both features and additional features.

[0061] At 230, the AR anchor point pose manager maintains the 3D size and dimensions of the features based on a model that includes the size and dimensions of real-world objects. In this embodiment, the model is a CAD model of the real-world object, and the CAD model includes the size and dimensions of the features, as well as other features of the real-world object.

[0062] In embodiments 221 and 230, at 231, the AR anchor pose manager maintains the size and dimensions of the features, as well as the additional size and dimensions of extra features, within a 3D array. The size and dimensions may also include different possible orientations of the features and additional features of real-world objects.

[0063] In embodiment 231, and at 232, the AR anchor pose manager obtains internal camera parameters of the camera 132 supplying the video from the AR app 135. These internal camera parameters may include the camera's focal length, lens distortion, etc.

[0064] At position 240, the AR anchor point pose manager estimates the position and orientation of the real-world object depicted in the video based on pixel coordinates, size, and dimensions. This is based on the position of features within the screen where the video is presented, as well as the known size and dimensions of features obtained from the model.

[0065] In embodiments 232 and 240, at 241, the AR anchor pose manager uses 2D arrays, 3D arrays, and internal camera parameters to process the PnP algorithm to estimate the position and orientation of real-world objects within the video. That is, when a real-world object is oriented within the physical environment, its origin is estimated as its position within the video, and its orientation is estimated within the video.

[0066] In one embodiment, at 250, the AR anchor pose manager (210-240) is processed as a processing framework for AR app 135 to track and update the position and orientation of real-world objects within the video, serving as anchor objects for the AR session. In another embodiment, at 251, the AR anchor pose manager processing framework is used to track and update the position and orientation associated with AR-generated and rendered objects during an AR session representing AR app 135, the positions and orientations of which overlap with the anchor objects within the video. In yet another embodiment, at 252, the AR anchor pose manager provides the framework to AR app 135 via API 136.

[0067] Figure 3This is a flowchart of another method 300 for anchor object pose resolution during an AR session, according to an example embodiment. The software module implementing method 300 is referred to as an "AR tracking framework service." The AR tracking framework service is implemented as executable instructions programmed and residing in memory and / or a non-transitory computer-readable (processor-readable) storage medium, and executed by one or more processors of one or more devices. The processor of the device executing the AR tracking framework service is specifically configured and programmed to process the AR tracking framework service. The AR tracking framework service may access one or more network connections during its processing. The network connection may be wired, wireless, or a combination of wired and wireless.

[0068] In one embodiment, the device for performing the AR tracking framework service is cloud 110. In another embodiment, the device for performing the AR tracking framework service is server 110. In yet another embodiment, the device for performing the AR tracking framework service is AR provider server 120.

[0069] In this embodiment, the AR tracking framework service is all or a combination of methods 113, 124, 136, and / or 200. The AR tracking framework service is presented in accordance with the above description... Figure 2 Method 200 discusses a different and enhanced approach from a different perspective.

[0070] At position 310, the AR tracking framework service maintains the CAD model of the terminal. The terminal is a composite or complex real-world object that contains various modules or features associated with it.

[0071] At position 320, the AR tracking framework service receives an AR session initiation request from AR app 135. The request includes video of the physical environment, where the device is located within the physical environment.

[0072] In this embodiment, at point 321, the AR tracking framework service identifies the AR session initiation request as a maintenance or service action, which will be performed or is being performed by an engineer at the terminal. The AR session is an AR maintenance session. It should be noted that other types of AR sessions can also be identified, such as gaming sessions, shopping sessions, etc.

[0073] At 330, the AR tracking framework service identifies three or more features of the terminal from the video. In an embodiment, at 331, the AR tracking framework service selects three or more features from a plurality of features based on the visual attributes of the three or more features detected in the video. For example, the three or more features are selected because they are the smallest or most unique features detectable from the video.

[0074] At 340, the AR tracking framework service obtains the size and dimensions of three or more features from the CAD model. The size and dimensions may also include orientations that can be used for each of the features within the terminal, as defined in the CAD model.

[0075] In this embodiment, at point 341, the AR tracking framework service maintains the pixel coordinates, size, and dimensions of three or more features in both 2D and 3D arrays. This is discussed in detail above regarding system 100 and method 200.

[0076] At position 350, the AR tracking framework service obtains the internal camera parameters of camera 132, which provides real-time video of the physical environment. These internal camera parameters may include the camera's focal length, lens distortion values, etc.

[0077] At 360, the AR tracking framework service uses the pixel coordinates, size, and internal camera parameters of three or more features presented on the screen of device 130 (which includes camera 132) to estimate the position and orientation of the terminal within the video. At 320, the AR tracking framework service initializes the AR session of AR App 135 in response to an AR session initiation request.

[0078] In embodiments 341 and 360, at 361, the AR tracking framework service uses 2D array, 3D array, and internal camera parameters to process the PnP algorithm to estimate the position and orientation of the terminal within the video. In embodiment 361, and at 362, the AR tracking framework service tracks the video after the AR session is initiated and updates the position and orientation of the terminal within the video, as well as any AR-generated objects presented within the video, during the AR session.

[0079] In this embodiment, at position 370, the AR tracking framework service continuously repeats to position 360 during the AR session to track and update the location and orientation within the video. The AR tracking framework service also tracks and updates the location and orientation of AR-generated objects superimposed on the video during the AR session.

[0080] It should be understood that describing software in a specific form (e.g., components or modules) is for the purpose of aiding understanding only and is not intended to limit the way software implementing those functions can be architected or constructed. For example, while a module may be shown as a separate module, it may be implemented as source code, as an individual component, some but not all of these modules may be composable, or the described functionality may be implemented in software constructed in any other convenient manner.

[0081] Furthermore, although the software module is shown to run on a single piece of hardware, the software can span multiple processors or be distributed in any other convenient way.

[0082] The above description is illustrative and non-limiting. Many other embodiments will become apparent to those skilled in the art upon reviewing the above description. Therefore, the scope of the embodiments should be determined by referring to the full scope of the appended claims together with their equivalents.

[0083] In the foregoing description of the embodiments, various features have been grouped together into a single embodiment for the purpose of simplifying this disclosure. This method of disclosure should not be construed as reflecting more features of the claimed embodiments than those expressly stated in the claims. In fact, as reflected in the appended claims, the subject matter of the invention lies in fewer than all the features of a single disclosed embodiment. Therefore, the following claims are incorporated herein by reference, wherein each claim itself represents a separate exemplary embodiment.

Claims

1. A method for resolving anchor point location and orientation, comprising performing the following steps: During an AR session initiated within a physical environment containing real-world objects, features associated with the real-world objects are identified in the video of the AR app; Maintain the pixel coordinates of the feature within the video; The size and dimensions of the features are maintained based on a model that includes the size and dimensions of the real-world objects; as well as Based on the pixel coordinates, the size, and the dimensions, the position and orientation of the real-world object as depicted in the video are estimated.

2. The method according to claim 1, wherein the method performs any one of the following steps (1)-(3): (1) The method is processed into a framework for the AR app to track and update the position and orientation of the real-world object in the video as an anchor object for the AR session; (2) The method is processed into a frame for the AR app to track and update the position and orientation of the real-world object in the video as an anchor object of the AR session, and the frame is processed to track and update the position and orientation associated with the AR-generated object during the AR session, the AR-generated object overlapping in the video by the position and orientation of the AR app relative to the anchor object; as well as (3) The method is processed into a frame for the AR app to track and update the position and orientation of the real-world object in the video as an anchor object of the AR session, and the frame is processed to track and update the position and orientation associated with the AR-generated object during the AR session, the AR-generated object overlapping the AR app in the video with respect to the position and orientation of the anchor object, and the frame is provided to the AR app through an application programming interface.

3. The method according to claim 1, wherein the method performs any one of the following steps (1)-(2): (1) The identifier further includes identifying the features from the video based on known features associated with the real-world object; and (2) The identifier further includes selecting the feature from a plurality of known features associated with the real-world object.

4. The method of claim 1, wherein maintaining the pixel coordinates further comprises maintaining the pixel coordinates of the features of the real-world object and the additional pixel coordinates of additional features in a two-dimensional (2D) array.

5. The method according to claim 4, wherein the method performs any one of the following steps (1)-(3): (1) Maintaining the size further includes maintaining the size and dimensions of the features of the real-world object and the additional size and dimensions of the additional features in a three-dimensional (3D) array; (2) Maintaining the size further includes maintaining the size and dimensions of the features of the real-world object and the additional size and dimensions of the additional features in a three-dimensional (3D) array, and estimation further includes obtaining internal camera parameters from the AR app for the camera supplying the video; and (3) Maintaining the size further includes maintaining the size and dimensions of the features of the real-world object and the additional size and dimensions of the additional features in a three-dimensional (3D) array, and estimating further includes obtaining internal camera parameters of the camera used to supply the video from the AR app, and estimating further includes using the 2D array, the 3D array and the internal camera parameters to process a perspective n-point algorithm to estimate the position and orientation of the real-world object within the video.

6. A method for resolving anchor point location and orientation, comprising performing the following steps: Maintain the computer-aided design (CAD) model of the terminal; Receive an AR session initiation request from an augmented reality (AR) application (app), the AR app including video of a physical environment, wherein the terminal is located within the physical environment; Identify three or more features of the terminal from the video; Obtain the size and dimensions of the three or more features from the CAD model; Obtain the internal camera parameters of the camera supplying the video; and The position and orientation of the terminal within the video are estimated using the pixel coordinates of the three or more features presented on the screen on the device including the camera, the size, the dimensions, and the internal camera parameters, and the AR session of the AR app is initialized in response to the AR session initiation request.

7. The method according to claim 6, wherein the method performs any one of the following steps (1)-(4): (1) During the AR session, the use is repeated continuously to track and update the position and orientation of the terminal within the video, and during the AR session, the position and orientation of AR-generated objects within the video are tracked and updated; (2) Receiving further includes identifying the AR session initiation request as a maintenance or service action to be performed by an engineer at the terminal; (3) The identifier further includes selecting the three or more features from a plurality of features based on visual attributes of the three or more features detected in the video; as well as (4) Provide the method as a cloud-based Software as a Service (SaaS) to the AR app.

8. The method according to claim 6, wherein the method performs any one of the following steps (1)-(3): (1) Obtaining the size further includes maintaining the pixel coordinates, the size, and the dimensions of the three or more features in a two-dimensional (2D) array and a three-dimensional (3D) array; (2) Obtaining the size further includes maintaining the pixel coordinates, the size, and the dimensions of the three or more features in a two-dimensional (2D) array and a three-dimensional (3D) array, and further includes using the 2D array, the 3D array, and the internal camera parameters to process a perspective n-point (PnP) algorithm to estimate the position and orientation of the terminal within the video; and (3) Obtaining the size further includes maintaining the pixel coordinates, the size, and the dimensions of the three or more features in a two-dimensional (2D) array and a three-dimensional (3D) array, and further includes using the 2D array, the 3D array, and the internal camera parameters to process the perspective n-point (PnP) algorithm to estimate the position and orientation of the terminal within the video, and tracking the video after the AR session is initiated, and updating the position and orientation of the terminal within the video and any AR-generated objects presented within the video during the AR session.

9. A system comprising: A cloud server, comprising at least one processor and a non-transitory computer-readable storage medium; The non-transitory computer-readable storage medium includes executable instructions; When the executable instructions are provided to and executed by the non-transitory computer-readable storage medium to the at least one processor, the at least one processor causes the at least one processor to perform operations including the following: Maintain the computer-aided design (CAD) model of the terminal; Provide application programming interfaces (APIs) for augmented reality (AR) applications (apps); and Interacting with the AR app via the API to provide AR session initialization and AR session tracking based on features of the terminal detected in the video within the physical environment during an AR session, in the following manner: 1) mapping the pixel coordinates of the terminal's features to their position on the screen of the device displaying the video; 2) maintaining the size and dimensions of the features and their pixel coordinates obtained from the CAD model; 3) obtaining the internal camera parameters of the camera supplying the video; 4) using the pixel coordinates, the size, the dimensions, and the internal camera parameters to process a perspective n-point (PnP) algorithm to estimate the terminal's position and orientation within the screen displaying the video for the AR session initialization; and 5) updating the terminal's position and orientation and any AR-generated objects during the AR session representing the AR application.

10. The system according to claim 9, wherein the terminal is a point-of-sale (POS) terminal, a self-service terminal (SST), an automated teller machine, or a self-service machine.

Images