Entity tracking method and device based on independent feature data

Abstract

The present invention relates to an entity tracking method and device based on feature data independent of an augmented reality engine. A URI for feature data about an entity is generated according to a combination of an entity name and an AR engine name. The URI is used to obtain feature data of the entity that represents the features of the entity in a format that can be processed by the AR engine. The AR engine can be run based on the feature data obtained in this way to perform entity tracking in an augmented reality space, thereby enabling entity tracking based on feature data independent of the augmented reality engine.

Description

Technical Field

[0001] The present invention relates to a method and a device for tracking an entity in an augmented reality space. Background Art

[0002] Augmented Reality (AR) is a technology that overlaps virtual images with images in real space and displays them as one image. In the augmented reality space corresponding to the real space, there is at least one entity that needs to overlap and display the virtual image. Since the movement of entities in the real space needs to be reflected in the augmented reality space, entity tracking technology is a representative technology provided by augmented reality applications. This entity tracking is performed by an augmented reality engine called by the augmented reality application. The augmented reality engine is referred to as the "AR engine" below.

[0003] Entity tracking technology can be divided into technology for tracking two-dimensional images and technology for tracking three-dimensional objects. When the entity is equivalent to a two-dimensional image, the AR engine generates feature data representing the characteristics of the two-dimensional image based on the two-dimensional image, and uses the feature data to track the entity in the augmented reality space. When the entity is equivalent to a three-dimensional object, the AR engine generates feature data representing the characteristics of the three-dimensional object based on multiple images representing the three-dimensional object, and uses the feature data to track the entity in the augmented reality space. In this way, the algorithm for generating feature data will be different depending on whether it is two-dimensional image tracking or three-dimensional object tracking.

[0004] Augmented reality applications are divided into native applications and web applications. Native applications are applications developed based on SDK (Software Development Kit) provided by Google's Android, iOS and other platforms, and can only run on platforms that support the SDK. Web applications are applications written in web languages ​​such as HTML (Hypertext Markup Language), CSS (Client Side Script), and JS (Java Script), and can run on both Android and iOS.

[0005] Since these various augmented reality applications use different AR engines, and the format of feature data used by each AR engine is different, each augmented reality application needs to generate feature data in a format suitable for the AR engine it uses. Therefore, existing augmented reality applications include feature data generation algorithms that rely on augmented reality engines.

[0006] Since the algorithms for generating feature data and feature matching algorithms are different for each AR engine, some AR engines may have superior tracking performance for two-dimensional images or for specific shapes of entities compared to other AR engines, but have poor tracking performance for another type and shape of entities such as three-dimensional objects. Therefore, attempts are being made to develop augmented reality applications using multiple types of AR engines, but because augmented reality applications need to include multiple feature data generation algorithms for all AR engines used by them, there are problems such as increasing difficulty in developing augmented reality applications and increasing size of augmented reality applications. Summary of the invention

[0007] Technical issues

[0008] The object of the present invention is to provide an entity tracking method and device based on feature data independent of an augmented reality engine, which does not use a feature data generation algorithm that depends on an augmented reality engine, but acquires feature data independently of the augmented reality engine and runs the AR engine, thereby being able to perform tracking of entities in an augmented reality space. In addition, the object is to provide a computer-readable storage medium storing a program for executing the entity tracking method on a computer. Not limited to the technical problems described above, another technical problem can be derived through the following description.

[0009] Technical Solution

[0010] According to one aspect of the present invention, an entity tracking method based on feature data independent of an augmented reality engine includes: a step of generating a URI (Uniform Resource Identifier) ​​of feature data about the first entity based on a combination of a name of the first entity and a name of a first AR engine; a step of obtaining, using the URI of the feature data about the first entity, feature data of the first entity that represents features of the first entity in a format that can be processed by the first AR engine; and a step of running the first AR engine based on the acquired feature data of the first entity to perform tracking of the first entity in the augmented reality space.

[0011] The step of generating a URI for the characteristic data about the first entity is to generate the URI for the characteristic data about the first entity according to a combination of the host name of the resource server, the name of the first entity and the name of the first AR engine. In the step of obtaining the characteristic data of the first entity, the URI for the characteristic data about the first entity can be used to request the characteristic data of the first entity from the resource server corresponding to the host of the URI for the characteristic data about the first entity, so as to obtain the characteristic data of the first entity from the resource server.

[0012] The entity tracking method also includes the step of generating a URI of the first entity based on a combination of the host name of the resource server and the name of the first entity; and the step of detecting a first AR engine to be used for tracking the first entity from among the multiple types of AR engines. In the step of generating a URI of feature data about the first entity, the URI of the feature data about the first entity can be generated by appending the name of the detected first AR engine to the generated URI of the first entity.

[0013] The entity tracking method also includes a step of detecting an entity tracking method that the first AR engine will use for the first entity. In the step of generating a URI for feature data about the first entity, the name of the detected first AR engine and the name of the detected entity tracking method can be added to the generated URI of the first entity to generate the URI for the feature data about the first entity.

[0014] In the step of detecting the entity tracking method, the entity tracking method to be used by the first AR engine for the first entity may be detected according to whether the first entity is a two-dimensional image or a three-dimensional object.

[0015] The entity tracking method also includes a step of detecting at least one entity in the augmented reality space represented by the augmented reality content from the augmented reality content. In the step of generating a URI for feature data about the first entity, the URI for the feature data about the first entity can be generated from a combination of the name of the first entity in the at least one detected entity and the name of the first AR engine.

[0016] The entity tracking method may include: a step of generating a URI for feature data about the second entity based on a combination of the name of the second entity and the name of the second AR engine; a step of acquiring feature data of the second entity representing the features of the second entity in a format that can be processed by the second AR engine using the URI for the feature data about the second entity; and a step of running the second AR engine based on the acquired feature data of the second entity to perform tracking of the second entity in an augmented reality space.

[0017] The entity tracking method also includes: a step in which the first AR engine confirms whether feature data or an original image of the first entity is needed in order to track the first entity; in the step of generating a URI for the feature data of the first entity, a URI for the feature data of the first entity may be generated if it is confirmed that the first AR engine needs the feature data of the first entity.

[0018] The entity tracking method may further include: when it is confirmed that the first AR engine needs the original image of the first entity, a step of generating a URI for the original image of the first entity according to a combination of a host name of a resource server and a name of the first entity; a step of obtaining the original image of the first entity using the URI for the original image of the first entity; and a step of running the first AR engine based on the obtained original image of the first entity to perform tracking of the first entity in the augmented reality space.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, storing a program for executing the entity tracking method on a computer.

[0020] According to another aspect of the present invention, an entity tracking device based on feature data independent of an augmented reality engine includes: a URI processing unit that generates a URI about feature data of a first entity based on a combination of a name of the first entity and a name of a first AR engine; and an AR processing unit that uses the URI about the feature data of the first entity to obtain feature data of the first entity that represents the features of the first entity in a format that can be processed in the first AR engine, and runs the first AR engine based on the acquired feature data of the first entity to perform tracking of the first entity in the augmented reality space.

[0021] Technical Effects

[0022] A URI for feature data about the entity is generated based on a combination of the name of the entity and the name of the AR engine. The URI is used to obtain feature data of the entity that represents the features of the entity in a format that can be processed by the AR engine. The AR engine is run based on the feature data thus obtained, thereby enabling tracking of the entity in the augmented reality space. Therefore, entity tracking can be performed based on the feature data independent of the augmented reality engine.

[0023] In this way, the feature data of the entity is obtained using the URI independently of the augmented reality engine, and the AR engine is run based on its feature data, so that the entity in the augmented reality space can be tracked, so the augmented reality application does not need to include a feature data generation algorithm that depends on the augmented reality engine. That is, the augmented reality application includes a feature data acquisition algorithm that is independent of the augmented reality engine, and therefore does not need to include a feature data generation algorithm that depends on the augmented reality engine.

[0024] When an augmented reality application uses multiple AR engines, it is not necessary to include multiple algorithms for generating feature data based on the original image of the entity for each AR engine, so that an efficient augmented reality application can be developed. The entity tracking in the augmented reality space according to the present invention can be achieved by using the URI of the feature data of the entity to obtain the feature data of the entity and only calling and running the AR engine based on the feature data, so that a general augmented reality application that does not include a feature data generation algorithm that depends on the AR engine can be provided.

[0025] The augmented reality application does not need to include a feature data generation algorithm for each AR engine, so the size of the augmented reality application can be reduced, thereby reducing the amount of work required to develop the augmented reality application. When developing an augmented reality application, most algorithms such as the algorithm for obtaining the feature data of an entity using the URI of the feature data of the entity can be reused intact in the existing augmented reality application, making the development of the augmented reality application very easy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a diagram showing the structure of an AR client 1 according to an embodiment of the present invention;

[0027] Figure 2 is a schematic diagram of an example of feature data of a two-dimensional image;

[0028] Figure 3 is a schematic diagram of an example of feature data about a three-dimensional object;

[0029] Figure 4 is a flow chart of an entity tracking method according to an embodiment of the present invention;

[0030] Figure 5 yes Figure 1 A schematic diagram of an example of a feature data acquisition process of the AR client 1 is shown;

[0031] Figure 6 is a schematic diagram of an operation example of multiple augmented reality applications according to another embodiment of the present invention;

[0032] Figure 7 is stored in Figure 1 A schematic diagram of an example of a URI table of resource server 2 is shown.

[0033] Description of Reference Numerals

[0034] 1: AR client 11: main processor

[0035] 12: AR processing unit 13: URI processing unit

[0036] 14: Communication module 15: Input module

[0037] 16: Output module 17: Camera module

[0038] 18: Sensor module 19: Memory

[0039] 20: Bus 2: Resource Server DETAILED DESCRIPTION

[0040] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments of the present invention to be described below relate to an entity tracking method and device based on feature data independent of an augmented reality engine, which can obtain feature data independently of an augmented reality engine and run an AR engine to perform entity tracking in an augmented reality space without using a feature data generation algorithm that depends on an augmented reality engine.

[0041] Hereinafter, the entity tracking method based on feature data independent of the augmented reality engine may be referred to as "entity tracking method", and the entity tracking device based on feature data independent of the augmented reality engine may be referred to as "entity tracking device" or "AR client". Hereinafter, the term "augmented reality" may be referred to as "AR", and "URI" is the abbreviation of "Uniform Resource Identifier".

[0042] Figure 1 FIG. 1 is a diagram showing the structure of an AR client 1 according to an embodiment of the present invention. Figure 1 According to the present embodiment, the AR client 1 is composed of a main processor 11, an AR processing unit 12, a URI processing unit 13, a communication module 14, an input module 15, an output module 16, a camera module 17, a sensor module 18, a memory 19 and a bus 20. According to the present embodiment, the AR client 1 tracks at least one entity existing in the augmented reality space, and then superimposes the tracked entity on the virtual image for display. Therefore, the AR client 1 according to the present embodiment can also be called an entity tracking device.

[0043] The main processor 11 executes an application, controls the AR processing unit 12, the URI processing unit 13, the input module 15, the output module 16, the communication module 14, the camera module 17, the sensor module 18 according to the program code of the application, and reads data from the memory 19 or stores data in the memory 19. In particular, the main processor 11 of this embodiment executes the augmented reality application described below. The main processor 11 may include, for example, a CPU (Central Processing Unit), an AP (Application Processor), and the like.

[0044] The AR processing unit 12 acquires, parses, and renders the augmented reality content as the augmented reality application runs. The URI processing unit 13 generates the URI of each entity, the URI of the characteristic data of an entity, and the URI of the original image of an entity as the augmented reality application runs. In this embodiment, the original image of an entity refers to the data representing a two-dimensional image or a three-dimensional image of the entity. The AR processing unit 12 and the URI processing unit 13 are processors that work as the augmented reality application runs. They can be physically the same processor as the main processor 11, or they can be another dedicated processor used by the main processor 11, such as a GPU (Graphics Processing Unit).

[0045] The communication module 14 communicates with the resource server 2 or other servers through a network such as the Internet. The communication module 14 can communicate with the resource server 2 or other servers through various communication methods such as LTE (Long Term Evolution) and wireless local area network (WiFi). The input module 15 receives user instructions, selections, information, etc. The input module 15 can include, for example, a touch panel, a keyboard, a mouse, etc. The output module 16 is composed of a combination of an image output device such as a display panel and an audio output device such as a speaker to provide visual and auditory output to the user.

[0046] The camera module 17 generates an image of the real space by photographing the space where the AR client 1 is located. The sensor module 18 uses the gyroscope, accelerometer, GPS (Global Positioning System) sensor, etc. built into the AR client 1 to detect the current position and posture of the AR client 1, thereby generating the current position and posture information of the AR client 1. The position information of the AR client 1 can be represented as the three-dimensional coordinate values ​​of x, y, and z in the augmented reality space corresponding to the three-dimensional coordinate system. The posture information of the AR client 1 can be represented as a roll value representing the rotation angle in the x-axis direction, a pitch value representing the rotation angle in the y-axis direction, and a yaw value representing the rotation angle in the z-axis direction.

[0047] The memory 19 stores the augmented reality application and the data used in its operation. Figure 1 In the example shown, the memory 19 stores augmented reality application A, AR engine A, AR engine B, augmented reality content, etc. Each AR engine is a function program called by the augmented reality application to perform entity tracking. The bus 20 is used to transfer data between the above-mentioned multiple components such as the main processor 11, the AR processing unit 12, and the URI processing unit 13.

[0048] The resource server 2 has a database of feature data hierarchically stored according to multiple entities, multiple AR engines, and multiple entity tracking methods using a URI system, and when receiving a certain URI, provides the corresponding feature data.

[0049] Figure 2 This is a schematic diagram of an example of feature data regarding a two-dimensional image. Figure 2 (a) shows a two-dimensional image of a house. Figure 2 (b) shows about Figure 2 The feature data of the two-dimensional image shown in (a). Figure 2 The cross point shown in (b) is equivalent to Figure 2 The feature points of the two-dimensional image shown in (a). The feature data about the two-dimensional image is binary data representing these feature points. Figure 2 (b) is to visualize the binary data in order to help understand the feature data of the two-dimensional image. Each AR engine can detect a portion of the feature data corresponding to the two-dimensional image from each of the multiple real space images continuously input by the camera module 17, and track each entity in the augmented reality space corresponding to the real space according to the change trend of the detected portion in the real space represented by the multiple real space images.

[0050] Figure 3 This is a schematic diagram of an example of feature data regarding a three-dimensional object. Figure 3 (a) shows any one of a plurality of two-dimensional images of a toy cat corresponding to a three-dimensional object. In order to represent a three-dimensional object, a plurality of two-dimensional images of the toy cat taken at multiple angles are required. Figure 3 (b) shows feature data generated based on a plurality of two-dimensional images of a toy cat. Figure 3 The points of the three-dimensional coordinate system shown in (b) correspond to the feature points of the toy cat corresponding to the three-dimensional object. The feature data for the three-dimensional image is binary data representing such feature points. Figure 3 (b) is to visualize the binary data in order to help understand the feature data about the three-dimensional object. Each AR engine can detect a portion of the feature data corresponding to the three-dimensional object from each of the multiple real space images continuously input by the camera module 17, and track each entity in the augmented reality space corresponding to the real space according to the change trend of the detected portion in the real space represented by the multiple real space images.

[0051] Figure 4 is a flow chart of an entity tracking method according to an embodiment of the present invention. Figure 4 According to the entity tracking method of this embodiment, Figure 1The AR client 1 shown is composed of steps of sequential processing. Therefore, even if the following contents are omitted, Figure 1 The contents of the AR client 1 described above are also applicable to the entity tracking method described below. The entity tracking method according to this embodiment can be implemented by an augmented reality application, and is executed by the main processor 11, the AR processing unit 12, the URI processing unit 13, etc. of the AR client 1. Figure 4 Each component of the AR client 1 , such as the main processor 11 , the AR processing unit 12 , and the URI processing unit 13 , will be described in detail.

[0052] In step 41, the main processor 11 of the AR client 1 receives an instruction from the user through the input module 15 to run the augmented reality application A according to this embodiment, and runs the augmented reality application A. Before running, the augmented reality application A is downloaded from the resource server 2 or other servers and installed on the AR client 1. The augmented reality application A can be provided by the resource server 2 or by other servers. This augmented reality application A can be written in a computer programming language such as Java. The following steps are performed by the AR processing unit 12 and the URI processing unit 13 as the augmented reality application A runs.

[0053] In step 42, the AR processing unit 12 of the AR client 1 displays information about a variety of augmented reality contents that can be provided by the augmented reality application A to the user through the output module 16. The user selects a certain augmented reality content that he wants to experience from the multiple augmented reality contents represented by the information displayed in this way. When the AR processing unit 12 receives user selection information for one of at least one augmented reality content that can be provided by the augmented reality application A through the input module 15, it obtains a certain augmented reality content represented by the user selection information from the resource server 2 or other servers through the communication module 14. The augmented reality content can be provided by the resource server 2 or other servers. The function of the communication module 14 is to send an augmented reality content request of the AR processing unit 12 to the resource server 2 or other servers through the network, and to receive augmented reality content from the resource server 2 or other servers and transmit it to the AR processing unit 12.

[0054] In step 43, the AR processing unit 12 of the AR client 1 parses the augmented reality content received in step 42, and detects at least one entity in the augmented reality space represented by the augmented reality content from the augmented reality content. The augmented reality content is content that represents each scene of the augmented reality space in a structured form by overlapping at least one virtual image on an image in the real space for display, and includes information about at least one entity existing in the augmented reality space, such as the name of each entity. In this way, the AR processing unit 12 can detect the name of each entity in the augmented reality space by parsing the augmented reality content.

[0055] Each entity existing in the augmented reality space is an element in which a virtual image corresponding to each entity is displayed overlappingly, and can be divided into a two-dimensional image and a three-dimensional object. Each entity existing in the augmented reality space displays a virtual image at its current position, so its position must be tracked. When an entity is a three-dimensional object, the virtual image displayed may be different depending on its current posture, so its posture can be tracked while tracking its position. In this embodiment, performing entity tracking refers to tracking the position of the entity or tracking the position and posture of the entity.

[0056] This augmented reality content can be written in a markup language such as XML (eXtensible Markup Language). The augmented reality content of this embodiment can represent a single frame image or multiple frames of images. Depending on the rendering mode, the augmented reality content of this embodiment can also be mixed reality content displayed as an augmented reality image or displayed as a virtual reality image.

[0057] In step 44, the URI processing unit 13 generates a URI (Uniform Resource Identifier) ​​of each entity based on a combination of the host name of the resource server 2 and the name of each entity detected in step 43. The URI processing unit 13 can generate the URI of each entity by adding the name of each entity detected in step 43 to the host name of the resource server 2. Taking the augmented reality content for displaying a virtual image that overlaps a house existing in a certain real space and represents the address of the house as an example, the URI processing unit 13 can generate the URI "http: / / server.com / house" of the entity by adding the entity name "house" to the host name "server.com" of the resource server 2.

[0058] Taking the augmented reality content for displaying a virtual image showing expression changes according to the weather in a certain real space by overlapping the face of a toy cat existing in a certain real space as an example, the URI processing unit 13 can generate the URI "http: / / server.com / cat" of the entity by adding the name of the entity "cat" to the host name "server.com" of the resource server 2. The host name "server.com" of the resource server 2 can be included in the augmented reality application and can also be included in the augmented reality content.

[0059] In step 45, the AR processing unit 12 selects an AR engine to be used for tracking each entity detected in step 41 from a plurality of types of AR engines. The AR processing unit 12 may detect the name of the AR engine to be used for tracking the entity from the setting information of the augmented reality application A or the parsing result of the augmented reality content in step 43. The augmented reality content may include the name of the AR engine that is most suitable for tracking each entity existing in the augmented reality space represented by it. The augmented reality application A may be designed so that the user can select any one of a plurality of types of AR engines. In this case, the user may select any one of a plurality of types of AR engines in the process of setting the operating environment of the augmented reality application A.

[0060] In step 46, the AR processing unit 12 determines which of the feature data and the original image of the entity that the AR engine detected in step 45 needs in order to track the entity detected in step 43, based on the attribute of the AR engine detected in step 45. In step 46, if it is confirmed that the AR engine needs the feature data of the entity, step 47 is executed. In step 46, if it is confirmed that the AR engine needs the original image of the entity, step 410 is executed. For example, when the attribute of AR engine A is set to always execute the process of generating its feature data based on a two-dimensional image before tracking the two-dimensional image, AR engine A needs two-dimensional image data instead of its feature data.

[0061] In step 47, the AR processing unit 12 detects the entity tracking method to be used by the AR engine detected in step 45 for each entity detected in step 43. More specifically, the AR processing unit 12 may detect the entity tracking method of the AR engine detected in step 45 to be used for each entity detected in step 43, depending on whether each entity detected in step 43 belongs to a two-dimensional image or a three-dimensional object. The two-dimensional image tracking method and the three-dimensional object tracking method in the augmented reality space vary depending on the data format between the two-dimensional image and the three-dimensional object.

[0062] In step 48, the URI processing unit 13 generates a URI for the feature data of each entity detected in step 43 by appending the name of the AR engine detected in step 45 and the name of the entity tracking method detected in step 47 to the URI of each entity generated in step 44. In this way, the URI processing unit 13 generates a URI for the feature data of each entity detected in step 43 based on a combination of the host name of the resource server 2, the name of each entity detected in step 43, the name of the AR engine detected in step 45, and the name of the entity tracking method detected in step 47.

[0063] Figure 5 yes Figure 1 FIG. 4 is a schematic diagram of an example of a feature data acquisition process of the AR client 1. For example, the name of the entity detected in step 43 is Figure 2 In the case where the name of the two-dimensional image is "house", the name of the AR engine detected in step 45 is "AREngineA", and the name of the entity tracking method detected in step 47 is "ImageTracking", Figure 2 The URI of the feature data of the two-dimensional image entity is "http: / / server.com / house / AREngineA / ImageTracking". For example, the name of the entity detected in step 43 is Figure 3 In the case where the name of the three-dimensional object is "cat", the name of the AR engine detected in step 45 is "AREngineB", and the name of the entity tracking method detected in step 47 is "ObjectTracking", Figure 3 The URI of the characteristic data of the three-dimensional object entity is "http: / / server.com / house / AREngineB / ObjectTracking".

[0064] The AR processing unit 12 and the URI processing unit 13 are processors that work as the augmented reality application runs, and they work according to the program code of the augmented reality application. Therefore, in order to help understand this embodiment, the augmented reality application is shown in the form of providing a URI and obtaining feature data.

[0065] In step 49, the AR processing unit 12 uses the URI of the feature data about each entity generated in step 49 to obtain the feature data of each entity that represents the features of each entity in a format that can be processed by the AR engine detected in step 45. The AR processing unit 12 uses the URI of the feature data about each entity generated in step 49 to access the resource server 2 of the host corresponding to the URI, request the feature data of each entity representing the features of each entity, and obtain the feature data of each entity as a response to the request from the resource server 2. The role of the communication module 14 is to send such a request of the AR processing unit 12 to the resource server 2 through the network, and receive a response to the request from the resource server 2 and pass it to the AR processing unit 12.

[0066] like Figure 5 As shown, about Figure 2 When the URI of the feature data of the two-dimensional image entity is "http: / / server.com / house / AREngineA / ImageTracking", the AR client 1 can use the URI to access the resource server 2 to obtain the Figure 2 The characteristic data of the two-dimensional image entity shown. Figure 5 As shown, about Figure 3 When the URI of the feature data of the three-dimensional object entity is "http: / / server.com / house / AREngineB / ObjectTracking", the AR client 1 can use the URI to access the resource server 2 to obtain the Figure 3 Characteristic data of the three-dimensional object entity shown.

[0067] In step 410, the URI processing unit 13 generates a URI for the original image of each entity detected in step 43 by adding "originalImage" to the URI of each entity generated in step 44. The original image of each entity refers to data used to represent a two-dimensional image or a three-dimensional image of each entity. The feature data of each entity has a very small data size compared to the original image of each entity, so that entity tracking can be effectively achieved.

[0068] In step 411, the AR processing unit 12 uses the URI of the original image of each entity generated in step 49 to obtain the original image of each entity detected in step 43. The AR processing unit 12 uses the URI of the original image of each entity generated in step 49 to access the resource server 2 and request the original image of each entity, and obtains the original image of each entity as a response to the request from the resource server 2. The function of the communication module 14 is to send such a request of the AR processing unit 12 to the resource server 2 through the network, and receive a response to the request from the resource server 2 and pass it to the AR processing unit 12. Among them, the original image of each entity refers to the data representing the two-dimensional image of each entity.

[0069] In step 412, the AR processing unit 12 runs the AR engine detected in step 45 based on the feature data obtained in step 49 or the original image obtained in step 411 to track each entity in the augmented reality space represented by the augmented reality content received in step 42. In the case where the AR engine detected in step 45 is run based on the original image obtained in step 411, the AR engine generates feature data of the original image based on the original image. The augmented reality space refers to the space in the real space where the AR client 1 is currently located that is to be enhanced and displayed using a virtual image through augmented reality content, which may be all or part of the real space.

[0070] The AR processing unit 12 detects a portion corresponding to the feature data of each entity from each of the multiple real space images continuously input from the camera module 17, and can track the position of each entity in the augmented real space according to the position change trend of the detected portion in the real space represented by the multiple real space images continuously input from the camera module 17. The AR processing unit 12 can also track the posture of each entity in the augmented real space according to the posture change trend of the detected portion in the real space represented by the multiple images continuously input from the camera module 17.

[0071] Next, the AR processing unit 12 generates an augmented reality image in which the virtual image of each entity is overlapped at the position of each entity tracked in step 412 for each of the multiple real space images input in real time from the camera module 17 by rendering the augmented reality content received in step 42. The AR processing unit 12 can also rotate and overlap the virtual image of each entity according to the posture change of each entity for each of the multiple real space images, referring to the position and posture of each entity tracked in step 412. The output module 16 displays the augmented reality image generated in this way. Figure 4 The illustrated embodiment relates to an entity tracking method, so in Figure 4 This step is omitted.

[0072] Figure 6 FIG. 1 is a schematic diagram of an operation example of multiple augmented reality applications according to another embodiment of the present invention. Figure 6 In the example shown, multiple augmented reality applications correspond one-to-one with multiple types of AR engines. In another example, one augmented reality application can also use multiple types of AR engines, and in another example, multiple augmented reality applications can also use one AR engine. Multiple augmented reality applications can also be set in one AR client 1, and can also be set in multiple AR clients 1 respectively.

[0073] After running the augmented reality application A, the entity detected is "entity A", and the attribute of the AR engine A is set to run based on the feature data of the entity A, and the entity A is tracked by executing steps 41 to 49 and 412. That is, the URI processing unit 13 generates a URI for the feature data of the entity A based on the combination of the host name of the resource server 2, the name of the entity A in the at least one entity detected in step 43, the name of the AR engine A detected in step 45, and the name of the entity tracking method detected in step 47. Afterwards, the AR processing unit 12 uses the URI to obtain feature data that represents the features of the entity A in a format that can be processed by the AR engine A.

[0074] Next, the AR processing unit 12 runs the AR engine A based on the feature data thus obtained to track the entity A in the augmented reality space. The AR processing unit 12 and the URI processing unit 13 are processors that work as the augmented reality application runs, and they work according to the program code of the augmented reality application. Therefore, in order to help understand this embodiment, the augmented reality application provides a URI and obtains feature data in a form shown.

[0075] As described above, each AR engine is a function program that performs entity tracking independently of the augmented reality application, and the AR processing unit 12 runs the AR engine A by calling the AR engine A and inputting the feature data of the entity A therein. Afterwards, the AR engine A returns the entity tracking result of the AR engine A to the AR processing unit 12, for example, returning the position of the entity A to the AR processing unit 12. Afterwards, the AR processing unit 12 overlaps the virtual image of each entity A at the position of the entity A for each of the multiple real space images input in real time from the camera module 17, thereby generating an augmented reality image for each real space image.

[0076] When the entity detected by running the augmented reality application B is "entity B" and the attribute of the AR engine B is set to run based on the feature data of the entity B, the tracking of the entity B is achieved by executing steps 41 to 49 and 412. That is, the URI processing unit 13 generates a URI for the feature data of the entity B based on a combination of the host name of the resource server 2, the name of the entity B in the at least one entity detected in step 43, the name of the AR engine B detected in step 45, and the name of the entity tracking method detected in step 47. Afterwards, the AR processing unit 12 uses the URI to obtain feature data that represents the features of the entity B in a format that can be processed by the AR engine B.

[0077] Next, the AR processing unit 12 runs the AR engine B based on the feature data thus obtained to track the entity B in the augmented reality space. More specifically, the AR processing unit 12 calls the AR engine B and inputs the feature data of the entity B to it and executes the AR engine B. Afterwards, the AR engine B returns the entity tracking result obtained by the AR engine B to the AR processing unit 12, for example, returns the position of the entity B to the AR processing unit 12. Afterwards, the AR processing unit 12 overlaps the virtual image of each entity B at the position of the entity B for each of the multiple real space images input in real time from the camera module 17, thereby generating an augmented reality image for each real space image.

[0078] When the entity detected by running the augmented reality application N is "entity N" and the attribute of the AR engine N is set to run based on the original image of the entity N, the entity N is tracked by executing steps 41 to 46 and 410 to 412. That is, the URI processing unit 13 generates a URI for the original image of the entity N based on the combination of the host name of the resource server 2 and the name of the entity N in at least one entity detected in step 43. Afterwards, the AR processing unit 12 obtains the original image of the entity N using the URI.

[0079] Next, the AR processing unit 12 tracks the entity N in the augmented reality space by executing the AR engine N based on the original image thus acquired. More specifically, the AR processing unit 12 runs the AR engine N by calling the AR engine N and inputting the original image of the entity N into it. Afterwards, the AR engine N returns the entity tracking result obtained by the AR engine N to the AR processing unit 12, for example, returns the position of the entity N to the AR processing unit 12. Afterwards, the AR processing unit 12 overlaps the virtual image of each entity N at the position of the entity N for each of the multiple real space images input in real time from the camera module 17, thereby generating an augmented reality image about the real space image. The operation process of other augmented reality applications will also be implemented according to any of the above two examples.

[0080] Figure 7 is stored in Figure 1 A schematic diagram of an example of a URI table of resource server 2 is shown. Figure 7 The URI table shown is a database of feature data that is hierarchically stored according to multiple entities, multiple AR engines, and multiple entity tracking methods using the URI system, and is visualized to help facilitate its understanding. Figure 7 Each URI in the URI table indicates where the feature data or original image of an entity is stored in the database of the resource server 2, so the AR client 1 can quickly obtain the feature data or original image of each entity. Figure 7 As shown in the URI table, multiple types of AR engines are classified into "ARCore", "ARKit", "Vuforia", and "Wikitude".

[0081] As described above, when the properties of an AR engine are set to always perform the process of generating feature data of an entity based on its original image before tracking the entity, the AR engine requires the original image rather than the feature data. In this case, if the name of the entity detected in step 43 is "house", the URI of the original image of the entity can be "http: / / server.com / outdoor / house / OriginalImage". If the name of the entity detected in step 43 is "cat", the URI of the original image of the entity can be "http: / / server.com / indoor / cat / OriginalImage".

[0082] The original image of an entity is the same regardless of the type of AR engine, so from the perspective of efficient use of the data storage space of the resource server 2, it is preferred to unify the URI of the original image for each entity. When the number of entities managed by the resource server 2 is very large, in order to quickly access the storage location of the feature data or original image of each entity, the URI of each entity can be hierarchically defined by grouping multiple entities. That is, the URI of an entity can be generated by appending the entity name to the name of the group to which the entity belongs.

[0083] ARCore is a native AR engine that can only run on Google's Android platform. It provides the "Motion tracking" function that estimates the camera position in real space, the "Environment understanding" function that estimates the collection system of real space, and the "Augmented images" function that tracks two-dimensional images. ARCore does not provide the function of tracking three-dimensional objects.

[0084] Figure 2In the case where the two-dimensional image shown is the entity detected in step 43, the URI of the entity generated in step 44 is "http: / / server.com / outdoor / house". In the case where the name of the AR engine detected in step 45 is "ARCore" and the name of the entity tracking method detected in step 47 is "AugmentedImages", the URI of the feature data generated in step 48 is "http: / / server.com / outdoor / house / ARCore / AugmentedImages". This example belongs to the case where the properties of "ARCore" are set to run based on the feature data of the entity. If the properties of "ARCore" are set to run based on the original image of the entity, the URI of the original image as described above can be used.

[0085] "ARKit" is a native AR engine that can only run on Apple's iOS platform. It provides the "ARWorldTrackingConfiguration" function to estimate the camera position in real space, the "ARGeoTrackingConfiguration" function to estimate the camera position using GPS, the "ARImageTrackingConfiguration" function to track two-dimensional images, and the "ARObjectScanningConfiguration" function to track three-dimensional objects.

[0086] Figure 2 In the case where the two-dimensional image shown is the entity detected in step 43, the URI of the entity generated in step 44 is "http: / / server.com / outdoor / house". In the case where the name of the AR engine detected in step 45 is "ARKit" and the name of the entity tracking method detected in step 47 is "ARImageTracking", the URI of the feature data generated in step 48 is "http: / / server.com / outdoor / house / ARKit / ARImageTracking".

[0087] Figure 3In the case where the three-dimensional object shown is the entity detected in step 43, the URI of the entity generated in step 44 is "http: / / server.com / indoor / cat". In the case where the name of the AR engine detected in step 45 is "ARKit" and the name of the entity tracking method detected in step 47 is "ARObjectScanning", the URI of the feature data generated in step 48 is "http: / / server.com / indoor / cat / ARKit / ARObjectScanning". These two examples belong to the case where the properties of "ARKit" are set to operate based on the feature data of the entity. If the properties of "ARKit" are set to operate based on the original image of the entity, the URI of the original image as described above can be used.

[0088] "Vuforia" is an AR engine from PTC that can run on both Android and iOS. It provides the "Area Targets" function that estimates the camera position in real space, the "ImageTargets" function that tracks two-dimensional images, and the "Model Targets" function that tracks three-dimensional objects.

[0089] Figure 2 In the case where the two-dimensional image shown is the entity detected in step 43, the URI of the entity generated in step 44 is "http: / / server.com / outdoor / house". In the case where the name of the AR engine detected in step 45 is "Vuforia" and the name of the entity tracking method detected in step 47 is "ImageTargets", the URI of the feature data generated in step 48 is "http: / / server.com / outdoor / house / Vuforia / Image Targets".

[0090] Figure 3In the case where the three-dimensional object shown is the entity detected in step 43, the URI of the entity generated in step 44 is "http: / / server.com / indoor / cat". In the case where the name of the AR engine detected in step 45 is "Vuforia" and the name of the entity tracking method detected in step 47 is "Model Targets", the URI of the feature data generated in step 48 is "http: / / server.com / indoor / cat / Vuforia / ModelTargets". These two examples belong to the case where the properties of "Vuforia" are set to run based on the feature data of the entity. If the properties of "Vuforia" are set to run based on the original image of the entity, the URI of the original image as described above can be used.

[0091] "Wikitude" is an AR engine from Wikitude that can be run on both Android and iOS. It provides a "Scene Tracking" function that estimates the camera position in real space, an "Instant Tracking" function that can recognize planes and overlay virtual images, an "Image Tracking" function that tracks two-dimensional images, and an "Object Tracking" function that tracks three-dimensional objects.

[0092] Figure 2 In the case where the two-dimensional image shown is the entity detected in step 43, the URI of the entity generated in step 44 is "http: / / server.com / outdoor / house". In the case where the name of the AR engine detected in step 45 is "Wikitude" and the name of the entity tracking method detected in step 47 is "ImageTracking", the URI of the feature data generated in step 48 is "http: / / server.com / outdoor / house / Wikitude / ImageTracking".

[0093] Figure 3In the case where the three-dimensional object shown is the entity detected in step 43, the URI of the entity generated in step 44 is "http: / / server.com / indoor / cat". In the case where the name of the AR engine detected in step 45 is "Wikitude" and the name of the entity tracking method detected in step 47 is "ObjectTracking", the URI of the feature data generated in step 48 is "http: / / server.com / indoor / cat / Wikitude / ObjectTracking". These two examples belong to the case where the properties of "Wikitude" are set to run based on the feature data of the entity. If the properties of "Wikitude" are set to run based on the original image of the entity, the URI of the original image as described above can be used.

[0094] As mentioned above, depending on the type of AR engine, not only are the functions it provides different, but the entity tracking performance of each AR engine is also different depending on whether the entity belongs to a two-dimensional image or a three-dimensional object, and the different entity tracking performance of each AR engine is presented according to the shape of the entity. Therefore, it is necessary to design augmented reality applications to provide users with what kind of augmented reality services are suitable for their AR engines based on the augmented reality content. In order for augmented reality applications to provide high-quality augmented reality services, multiple different AR engines can also be used. In particular, "ARCore" and "ARKit" can only run on their dedicated platforms, so it is necessary to design respective augmented reality applications for the same augmented reality service.

[0095] According to this embodiment, independent of the AR engine and the entity tracking method, the feature data of the entity is obtained from the resource server 2 using the URI of the feature data about the entity, and the AR engine is run based on the feature data thus obtained. Therefore, the augmented reality application does not need to include multiple algorithms for generating feature data according to the original image of the entity according to the AR engine and the entity tracking method, so that efficient augmented reality applications can be developed.

[0096] According to the present embodiment, the tracking of entities in the augmented reality space can be achieved by simply obtaining the feature data of the entity from the resource server 2 using the URI of the feature data about the entity and calling to run AR, thereby providing a general augmented reality application that does not include a feature data generation algorithm that depends on the AR engine and the entity tracking method.

[0097] The augmented reality application does not need to include a feature data generation algorithm according to the AR engine and the entity tracking method, so that the size of the augmented reality application can be reduced, thereby reducing the amount of work required to develop the augmented reality application. In the case of developing an augmented reality application, most algorithms such as an algorithm for obtaining feature data of an entity from a resource server 2 using a URI of feature data about the entity can be directly reused in the existing augmented reality application, thereby making the development of the augmented reality application very easy.

[0098] In addition, the entity tracking method according to one embodiment of the present invention as described above can be written into a program that can be run on a processor of a computer, and can be recorded using a computer-readable storage medium and implemented on a computer running the program. Computers include all types of computers that can run programs, such as desktops, laptops, smart phones, and embedded computers. In addition, the data structure used for one embodiment of the present invention described above can be recorded on the computer-readable storage medium by various means. The computer-readable storage medium includes storage media such as random access memory (RAM), read-only memory (ROM), magnetic storage media (e.g., floppy disks, hard disks, etc.), optically readable media (e.g., CD-ROMs, DVDs, etc.).

[0099] The present invention is described above with the preferred embodiments as the center. A person skilled in the art of the present invention should understand that the present invention can be implemented in a modified manner without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be understood from the perspective of description rather than from the perspective of limitation. The scope of the present invention is not shown in the above description, but the scope of the rights, and all differences within the equivalent scope should be interpreted as included in the present invention.

Claims

1. An entity tracking method is an entity tracking method based on feature data independent of an augmented reality engine, characterized in that: include: A step of generating a uniform resource identifier URI of feature data about the first entity according to a combination of a name of the first entity and a name of the first AR engine; A step of acquiring, by using a URI of feature data about the first entity, feature data of the first entity representing the features of the first entity in a format that can be processed by the first AR engine; as well as The first AR engine is run based on the acquired feature data of the first entity to perform the step of tracking the first entity in the augmented reality space.

2. The entity tracking method according to claim 1, characterized in that: The step of generating the URI of the feature data about the first entity is to generate the URI of the feature data about the first entity according to a combination of a host name of a resource server, a name of the first entity, and a name of the first AR engine, The step of obtaining the characteristic data of the first entity is to use the URI of the characteristic data of the first entity to request the characteristic data of the first entity from the resource server of the host corresponding to the URI of the characteristic data of the first entity, so as to obtain the characteristic data of the first entity from the resource server.

3. The entity tracking method according to claim 2, characterized in that: Before the step of generating a uniform resource identifier URI of feature data about the first entity according to a combination of the name of the first entity and the name of the first AR engine, the method further includes: A step of generating a URI of the first entity based on a combination of the host name of the resource server and the name of the first entity; and a step of detecting a first AR engine from a plurality of types of AR engines to be used for tracking the first entity, The step of generating the URI of the feature data about the first entity is to generate the URI of the feature data about the first entity by adding the name of the detected first AR engine to the generated URI of the first entity.

4. The entity tracking method according to claim 3, characterized in that: After the step of detecting a first AR engine to be used for tracking the first entity from a plurality of types of AR engines and before the step of generating a URI for feature data about the first entity, the method further includes: a step of detecting an entity tracking method to be used by the first AR engine for the first entity, The step of generating a URI for the feature data of the first entity is to add the name of the detected first AR engine and the name of the detected entity tracking method to the generated URI of the first entity to generate the URI for the feature data of the first entity.

5. The entity tracking method according to claim 4, characterized in that: The step of detecting the entity tracking method to be used by the first AR engine for the first entity is to detect the entity tracking method to be used by the first AR engine for the first entity according to whether the first entity belongs to a two-dimensional image or a three-dimensional object.

6. The entity tracking method according to claim 1, characterized in that: Also includes: a step of detecting, from augmented reality content, at least one entity in an augmented reality space represented by the augmented reality content, The step of generating the URI of the feature data about the first entity is to generate the URI of the feature data about the first entity according to a combination of the name of the first entity in the at least one detected entity and the name of the first AR engine.

7. The entity tracking method according to claim 1, characterized in that: Also includes: A step of generating a URI for feature data of the second entity according to a combination of a name of the second entity and a name of a second AR engine; A step of acquiring, by using a URI of feature data about the second entity, feature data of the second entity representing the feature of the second entity in a format that can be processed by the second AR engine; as well as The second AR engine is run based on the acquired feature data of the second entity to perform the step of tracking the second entity in the augmented reality space.

8. The entity tracking method according to claim 2, characterized in that: Before the step of generating a URI for the characteristic data of the first entity, the method further includes: The first AR engine determines whether feature data or an original image of the first entity is needed in order to track the first entity. The step of generating the URI for the feature data of the first entity is, when it is confirmed that the first AR engine needs the feature data of the first entity, generating the URI for the feature data of the first entity.

9. The entity tracking method according to claim 8, characterized in that: Also includes: When it is confirmed that the first AR engine needs the original image of the first entity, a step of generating a URI for the original image of the first entity according to a combination of a host name of a resource server and a name of the first entity; The step of acquiring the original image of the first entity by using the URI of the original image of the first entity; as well as The step of running the first AR engine based on the acquired original image of the first entity to perform tracking of the first entity in the augmented reality space.

10. A computer-readable storage medium, characterized in that: A program for executing the entity tracking method according to any one of claims 1 to 9 on a computer is stored.

11. An entity tracking device is an entity tracking device based on feature data independent of an augmented reality engine, characterized in that: include: a URI processing unit configured to generate a URI for feature data related to the first entity according to a combination of a name of the first entity and a name of the first AR engine; as well as An AR processing unit, which uses a URI to the feature data of the first entity to obtain the feature data of the first entity that represents the features of the first entity in a format that can be processed by the first AR engine, and runs the first AR engine based on the obtained feature data of the first entity to perform tracking of the first entity in an augmented reality space.

Images