Method and apparatus for scalable semantic aware augmented reality (AR) + internet system

Abstract

In accordance with an embodiment, a computing device performs objection detection on media content of a physical object to match features of the physical object. The computing device sends a spatial object name service (SONS) query request to a SONS server. The SONS query request comprises the characteristic of the physical object. The computing device receives a SONS query response from the SONS server. The SONS query response includes a domain name system (DNS) domain name for retrieving augmented reality (AR) content corresponding to the physical object, the DNS domain name based on the feature of the physical object; and the computing device obtains an Internet Protocol (IP) address by parsing the DNS domain name. And the computing device obtains the AR content corresponding to the physical object based on the IP address and the SONS query response. The computing device renders at least a portion of the AR content corresponding to the physical object.

Description

Technical Field

[0001] The present disclosure relates generally to Internet systems, and in certain embodiments, to techniques and mechanisms for a scalable semantically-aware augmented reality (AR) + Internet system. Background Art

[0002] The term metaverse refers to a collective, virtual and real shared space that is interactive, immersive and collaborative. The metaverse can accommodate many users who can interact with each other in real time. The metaverse is created through the fusion of virtual augmented physical reality and physical persistent virtual space, supporting its users to experience a sense of "concreteness" and "presence", making users feel that they are actually in a virtual space with others, seeing things from a first-person perspective, and most likely in three-dimensional (3-dimentaional, 3D) form.

[0003] The Metaverse can be thought of as augmented reality (AR) for the Internet. AR is a real-time, direct or indirect view of a physical, real-world environment whose elements are enhanced by computer-generated sensory input (e.g., sound, video, graphics, or GPS data). The AR-based Internet can have many applications, such as AR advertising, immersive collaboration in design, manufacturing, and remote assistance, education, gaming, and meta-commerce (e.g., non-fungible tokens (NFTs), digital assets, and immersive applications).

[0004] Over time, AR technology has evolved from registering simple AR content in a physical environment (e.g., a table) for a single user to more complex platform systems that can include content creation, AR cloud, AR map, AR portal, AR contextual computing, and AR content management. The current system is limited by the lack of technology that integrates digital and physical space information representation. The current system is further limited by the lack of support for intelligent, multi-sensory, naturalized active content presentation, indexing, storage, discovery, and retrieval. The current system also lacks a standard format for 3D Internet information representation. Therefore, a more efficient resource locator solution, a more efficient AR Internet communication protocol, and a more efficient information communication, delivery, and presentation solution are needed to achieve a robust, secure, and scalable system solution for the AR Internet (e.g., Internet-based AR services). Summary of the invention

[0005] Technical advantages are generally achieved through the embodiments of the present disclosure, which describe methods and devices for scalable semantically-aware Internet-based AR services (which may be referred to as AR+Internet in this disclosure).

[0006] According to an embodiment, a computing device performs objection detection on the media content of a physical object to match the characteristics of the physical object. The computing device sends a SONS query request to a space object name service (SONS) server. The SONS query request includes the characteristics of the physical object. The computing device receives a SONS query response from the SONS server. The SONS query response includes a domain name system (DNS) domain name for retrieving augmented reality (AR) content corresponding to the physical object, and the DNS domain name is based on the characteristics of the physical object. The computing device obtains an Internet Protocol (IP) address by resolving the DNS domain name. The computing device obtains the AR content corresponding to the physical object based on the IP address and the SONS query response. The computing device renders at least a portion of the AR content corresponding to the physical object. A portion of the AR content may correspond to a first portion of the physical object. A portion of the AR content may be mixed (e.g., visually mixed) with a second portion of the physical object.

[0007] In some embodiments, the media content of the physical object may include at least one of one or more images of the physical object, audio of the physical object, or text of the physical object.

[0008] In some embodiments, performing objection detection may include: inputting media content into a trunk classification engine to obtain trunk features from the trunk classification engine; inputting trunk features into a neck classification engine to obtain neck features from the neck classification engine; and inputting neck features into a head classification engine to obtain head features from the head classification engine.

[0009] In some embodiments, the trunk classification engine may include a trunk neural network. The neck classification engine may include a neck neural network. The head classification engine may include a head classification neural network. In some embodiments, the features of the physical object in the SONS query request sent to the SONS server may include trunk features, neck features, and head features. In some embodiments, the SONS server may generate a uniform resource locator (URL) based on a combination of trunk features, neck features, and head features. The head features may include mapping information from media content to semantic information of the physical object, which is associated with the URL. In some embodiments, the mapping information may be stored in a block of a blockchain.

[0010] In some embodiments, the features of the physical object may correspond to the body of the physical object.

[0011] The embodiments in the present disclosure provide an AR+ Internet and an AR+ browser, which, combined with related tools and services, naturally supports active content presentation, indexing, storage, discovery, and retrieval. These embodiments utilize and improve existing decentralized designs and AI technologies. One focus of the embodiments is a real-world naming service mapped to the metaverse (i.e., the DNS+ of the physical world). The embodiment naming service supports multi-level, decentralized, and naturalized intelligent naming services for the physical world. The embodiment naming service also corresponds to a multi-level media classification based on ontology. The embodiment naming service also supports joint cloud and edge caching and processing for decentralized AR content creation, storage, and access. The embodiment naming service also supports a content-addressable Internet based on blockchain, with multi-sensory content creation and multi-sensory content access. The embodiments in the present disclosure support dynamic content discovery and driving for personalized and interactive active content presentation, multi-modal (visual, audio, tactile, etc.) content presentation, and a semantically aware AR+ browser engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 A diagram showing how AR is evolving with the Internet;

[0014] Figure 2 is a signal sequence diagram of a workflow using a Spatial Object Name Service (SONS) according to some embodiments;

[0015] Figure 3 shows a scalable image classification network architecture according to some embodiments;

[0016] Figure 4A shows an example of an image category ontology according to some embodiments;

[0017] Figure 4B shows an example of a media category ontology according to some embodiments;

[0018] Figure 5 shows an AR+Cloud architecture 500 according to some embodiments;

[0019] Fig. 6A shows the use of some embodiments Figure 5 The architecture shown builds an exemplary process of AR+Internet;

[0020] Figure 6B shows the use of some embodiments Figure 5 The architecture shown builds an exemplary system of AR+Internet;

[0021] Fig. 7A An embodiment of an AR+Internet Domain Name Service, Spatial Object Name Service (SONS), is shown according to some embodiments;

[0022] Figure 7B illustrates exemplary SONS spatial object identifier (SOid) and SONSLink generation and retrieval according to some embodiments;

[0023] Figure 7C shows SONS storage and retrieval based on an embodiment blockchain according to some embodiments;

[0024] Fig.7D shows an exemplary SONS hierarchy according to some embodiments;

[0025] Fig. 7E An exemplary system in AR+Internet according to some embodiments is shown;

[0026] Figure 8 A flowchart illustrating a method of retrieving AR content corresponding to a physical object using SONS according to some embodiments;

[0027] Fig. 9 A block diagram of a computing system that may be used to implement the devices and methods disclosed herein is shown, according to some embodiments.

[0028] Corresponding numerals and symbols in the different figures generally refer to corresponding parts unless otherwise indicated.The drawings are for the purpose of clearly illustrating the relevant aspects of the embodiments and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0029] The making and using of the disclosed embodiments are discussed in detail below. However, it should be understood that the concepts disclosed herein can be embodied in a wide range of specific contexts, and the specific embodiments discussed herein are merely illustrative and are not intended to limit the scope of the claims. Furthermore, it should be understood that various changes, substitutions, and modifications may be made herein without departing from the spirit and scope of the present disclosure as defined by the appended claims.

[0030] Figure 1 A functional representation of how AR functionality and systems have evolved with the Internet is shown. System 102 shows AR with a traditional World Wide Web (WWW). System 102 includes fixed content that is completely creator-generated and creator-managed. Also in system 102, the presentation of fixed content is managed by the creator. The content is presented in a two-dimensional (2D) web page with minimal interactive accessibility and controllability.

[0031] System 104 shows a system having WWWAR In system 104, content is created by creators and presentation of content is driven by consumer users. In system 104, content can be presented in 3D or 4D (4-dimentional, 4D) form with enhanced interactive accessibility and controllability. The browser in system 104 can naturally support active content presentation.

[0032] In order to better support the development of AR on the Internet, the disclosed embodiments describe the use of AR+Internet to build a better metaverse. AR+Internet refers to a network that has the ability to provide AR services for virtual spaces that are integrated with physical spaces among other services and functions provided by the current Internet, and the physical space can be accessed in a dual-space browser. In other words, AR+Internet is an Internet that combines virtual shared space with physical space, which is easy to achieve integration across application domains and platforms, thereby enhancing user experience and being applied to new businesses and applications. AR+Internet is combined with the disclosed blockchain-based spatial object name service (SONS), ontology-based media classification, joint cloud and edge processing, so that the content-addressable Internet naturally supports dynamic content discovery and driving to achieve personalized and interactive active content presentation, and supports decentralized AR content creation, storage and access. These features in turn provide the foundation and building blocks for creating a viable ecosystem in the AR+Internet.

[0033] One goal of the AR+Internet is to build a new dual-space (e.g., virtual and physical) network combined with a new browser (e.g., AR+ Browser) that provides: dynamic content creation by creators, scalable driven presentation by consumers to meet personalized preferences and application needs, natural user interfaces with enhanced accessibility and controllability of interactive content, cross-platform interoperability, decentralization for improved scalability, and a decentralized Internet combined with decentralized websites, web applications, and storage systems. Therefore, building an AR+Internet for the Metaverse may involve leveraging and further modifying existing decentralized designs and artificial intelligence (AI) technologies.

[0034] In supporting the AR+Internet, a focus of the present disclosure is the real-world naming service mapped to the metaverse (i.e., the naming service for the physical world). Naming services can be important because naming services can be the basis for many important metaverse applications. Naming services are very important in content discovery (e.g., finding advertisements associated with physical objects detected in the physical world) for accurately identifying physical objects, landmarks, and the delivery of associated content. Naming services are also important in content delivery because the mapping, routing, streaming, and playback of content (which uses naming services) have certain performance requirement metrics. In addition, efficient naming services are an important component of low latency because many metaverse applications in the AR+Internet use immersive interactions.

[0035] Providing effective and efficient naming services that meet the above accuracy, performance, and latency requirements is technically challenging for current systems and technologies. The technical challenge may be because "naming the world" is difficult. From the perspective of metaverse applications, the physical world is large. For example, for products in the world, there may be thousands of categories, millions of brands, and billions of stock keeping units (SKUs).

[0036] Faced with such a large scale, performance requirements still need to be maintained in terms of reliability, accuracy, fast AR+ content delivery (e.g., less than 200ms), etc. In addition, content delivery needs to be secure to prevent unauthorized content access, modification, and delivery.

[0037] The embodiments of the AR+ Internet in the present disclosure integrate a blockchain-based spatial object name service (SONS), as well as a content-addressable network, a semantic web, and an ontology-based media (e.g., images, audio, and video) classification combined with a hierarchical SONS. The AR+ Internet is a network that merges virtual space with physical space, which is accessible in a dual-space browser (e.g., an AR+ browser). The AR+ Internet is combined with blockchain-based SONS, ontology-based media classification, and joint cloud and edge processing to enable content-addressable AR+ Internet to naturally support dynamic content discovery and drive to achieve personalized and interactive active content presentation, as well as decentralized AR content creation, storage, and access.

[0038] Figure 22 is a signal sequence diagram 200 of a workflow using SONS according to some embodiments. In operation 212, a user terminal device 204 (e.g., a mobile phone or a personal computer (PC) that can be connected to a virtual reality (VR) headset or VR glasses) performs feature extraction on a physical object (e.g., a coke can) in the physical world 202. The VR headset can be a head-mounted device that completely covers the eyes to achieve an immersive 3D experience. The VR glasses can be a head-mounted device that partially covers the eyes to achieve an immersive 3D experience, and the user can still see the physical world through the VR glasses. The user terminal device 204 can use computer vision techniques known in the art to perform feature extraction based on multimedia (e.g., one or more images) of the detected physical objects. The multimedia may come from a real-time recording of the physical world as part of an AR display to be mixed with the virtual world in the AR display. The user terminal device 204 can also use other technologies such as tags attached to physical objects or quick response (QR) codes for physical object detection and feature extraction.

[0039] In some embodiments, the user terminal device 204 may perform feature extraction using any of the backbone model-based object detection and classification systems known in the art. Modern object detection and classification systems may include multiple components: (1) a backbone for extracting features; (2) a neck for feature fusion (which may include fusion of feature pyramids); (3) a head for object classification and bounding box regression (object detection). Such backbone model-based detection and classification systems are known in the art and are implemented by those skilled in the art.

[0040] The head consumes the features generated by the neck. The user terminal device 204 can input the multimedia of the physical object into the backbone neural network to obtain the backbone features output by the backbone neural network. Then, the user terminal device 204 can input the backbone features into the neck neural network to obtain the neck features output by the neck neural network. Next, the user terminal device 204 can input the neck features into the head neural network to obtain the head features output by the head neural network. The backbone neural network and the neck neural network can be pre-trained neural networks downloaded by the user terminal device 204, or they can be system-level neural networks of the user terminal device 204. The head neural network can be an application-specific neural network.

[0041] In operation 214, the user terminal device 204 may send a SONS query request (e.g., a lookup request) to a spatial object name service (SONS) 206. SONS 206 may be a service including one or more SONS servers. The SONS query request may include the features of the detected physical object obtained from operation 212. In some embodiments, the features of the physical object in the SONS query request sent to the SONS server of SONS 206 may include a trunk feature, a neck feature, and a head feature.

[0042] In operation 216, the user terminal device 204 may receive a SONS query response from SONS 206. The SONS response may include a Domain Name System (DNS) domain name of the host storing the AR content. The AR content information in the SONS query response may also include an identifier of the AR content corresponding to the physical object. The AR content information in the SONS query response may be in the form of a uniform resource locator (URL). For example, the SONS query response may include " / dns / example.com / foo / bar / baz.png" as a URL. In some embodiments, the URL may include information indicating the category to which the corresponding physical object belongs (e.g., "SONLink / foo.example.sod / directory / image / category1 / sample1.jpg"). The domain name of the host in the URL is "example.com", and the identifier of the AR content is " / foo / bar / baz.png" (e.g., the path of the AR content).

[0043] The DNS domain name (e.g., URL) may be based on the characteristics of the physical object. For example, SONS 206 may store a mapping between AR content information (e.g., URL) and information related to the characteristics of the physical object in the blockchain. SONS 206 may use the information about the characteristics of the physical object as a search keyword to parse the AR content information (e.g., URL) to find the AR content information stored in the block of the blockchain.

[0044] In operation 218, the user terminal device 204 may obtain an Internet Protocol (IP) address of a host in a cloud system (e.g., cloud system 210) storing AR content by resolving a DNS domain name (e.g., "example.com"). For example, the user terminal device 204 may send a DNS lookup query to one or more DNS servers 208 in operation 218, and receive a resolved IP address in operation 220.

[0045] AR content may be stored in the cloud system 210. The cloud system 210 may be a distributed cloud storage system. For example, the cloud system 210 may be an interplanetary file system (IPFS). IPFS is a protocol and peer-to-peer network for storing and sharing data in a distributed file system. IPFS uses content addressing to uniquely identify each file (e.g., AR content) in a global namespace that connects all computing devices. AR content may be content to be mixed as part of an AR display presented to a user. For example, the physical object is a can of coke, and the AR content may be a coke advertisement to be played to the user as part of the AR display. After the user terminal device 204 obtains the IP address of the host in the cloud system 210, the user terminal device 204 may retrieve the AR content corresponding to the physical object based on the IP address of the host storing the AR content and the AR content information (e.g., URL) in the SONS query response. In operation 222, the user terminal device 204 may send an AR content retrieval request to the host in the cloud system 210. The AR content retrieval request may include an identifier of the AR content for the host to retrieve the AR content. In some embodiments, the AR content retrieval request may include an interPlanetary file system (IPFS) content address converted from a URL (e.g., " / ipfs / QmW98pJrc6FZ6 / dns / example.com / foo / bar / baz.png" converted from " / dns / example.com / foo / bar / baz.png"). In operation 224, the cloud system 210 may retrieve the AR content based on the AR content retrieval request. For example, the cloud system 210 may retrieve the AR content using the IPFS content address in the AR content retrieval request. The AR content may be transmitted from the host to the user terminal device 204 in an AR content retrieval response to be presented by the user terminal device 204. The user terminal device 204 may render at least a portion of the AR content corresponding to the physical object. A portion of the AR content may correspond to a first portion of the physical object. A portion of the AR content may be mixed (e.g., visually mixed) with a second portion of the physical object.

[0046] The AR content may be interactive content that supports AR content interaction between the user terminal device 204 and the physical object (e.g., the user picks up a can of soda or changes the perspective of the can of soda) in operation 226. Therefore, in response to the AR content interaction in operation 226, the host in the cloud system 210 may continue to transmit new AR content in operation 228.

[0047] Figure 3Scalable image classification network architecture according to some embodiments is shown. The backbone classification engine 302 takes multimedia (e.g., one or more images of a coke can) as input and uses basic features of different resolutions of the multimedia to output backbone (BB) features. The neck classification engine 304 fuses features of different resolutions (BB features) to output neck features. The head classification engine 306 takes neck features as input and performs detection and / or classification of physical objects at different resolutions by outputting head features and / or detection and / or classification results.

[0048] exist Figure 3 In some of the illustrated embodiments, the category model 312 and the backbone classification engine 302 using the category model 312 can be stored locally in the user terminal device. The neck model and the neck classification engine 304 using the neck model can be stored in multiple edge servers 314. Each of the multiple edge servers 314 can have its own copy of the neck model. The use of multiple edge servers with consistent hashing supports load balancing in model inference. Different physical object types (e.g., different product brands) have different corresponding authoritative brand models 316 used by the head classification engine 306. The authoritative brand model 316 and the head classification engine 306 can also be stored in a distributed manner in multiple edge servers 314 to achieve load balancing.

[0049] In some embodiments, the trunk classification engine 302, the neck classification engine 304, and the head classification engine 306 can be implemented as a trunk neural network, a neck neural network, and a head network, respectively, as known in the art. In other embodiments, the trunk classification engine 302, the neck classification engine 304, and the head classification engine 306 can be implemented using other types of trunk classification engines, neck classification engines, and head classification engines, respectively, as known in the art.

[0050] The backbone classification engine 302 and the head classification engine 306 can encode the input content (e.g., media content such as images) into high-dimensional feature vectors. Based on these high-dimensional feature vectors, the head classification engine 306 can decode features for specific tasks (e.g., object detection, image classification). In order to optimize edge node processing efficiency, it may be necessary to aggregate feature query traffic based on high-level ontology categories. For example, images including soda cans can be routed to specific edge cache servers optimized for soda product detection tasks, images including cars can be routed to specific edge cache servers optimized for car detection tasks, and so on. The edge cache server can feed high-dimensional features from the decoder to multiple head / decoding task networks. Each head can be optimized for a specific task (e.g., detecting Coca-Cola products). The output of the head can create a mapping from input content (e.g., media content such as images) to specific semantic information (e.g., a specific Coca-Cola product). Semantic information can be used to associate with a predefined URL.

[0051] Therefore, the scalable image classification network architecture described above supports combining e.g. Figure 4A The image category ontology shown and Figure 4B Ontology-based classification of media (e.g., images, audio, and video) of the media category ontology shown.

[0052] Figure 5 FIG. 5 shows an AR+ cloud architecture 500 according to some embodiments. The AR+ cloud architecture 500 supports a feature DB system 502, a feature registration system 504, and a feature extraction system 506. In the feature DB system 502, there is an on-chain DNS system (e.g., DNS+) 512 and an AR content off-chain cloud 514. The on-chain DNS system 512 can be a reference Figure 2 SONS206 described. AR content chain cloud 514 can be referenced Figure 2 A cloud system 210 is described for storing AR contents.

[0053] For the feature registration system 504, the owner of the physical object (e.g., the manufacturer of the Coke product, or the creator of the AR content corresponding to the Coke product) can create AR content corresponding to the physical object (e.g., an AR commercial for a Coke can). The owner of the physical object can extract the features of the physical object (e.g., the trunk features, neck features, and head features described above). The owner of the physical object can store the AR content in the AR content off-chain cloud 514, and register the AR content with the on-chain DNS system 512 using the extracted features of the physical object, so that the terminal device can later contact the on-chain DNS system 512 to find out how to locate the AR content (e.g., obtain a URL for locating the AR content).

[0054] For the feature extraction system 506, the feature extraction operation is performed on the user terminal device (for example, Figure 2 After operation 212 described in the above, the user terminal device may send a query request to the on-chain DNS system 512 (e.g., Figure 2 ), and receiving the identified URL for the AR content (e.g., as Figure 2 Using the identified URL, the user terminal can continue to retrieve and download the AR content from the AR content off-chain cloud 514 (e.g., as shown in FIG. Figure 2 Operations 218 to 228 described in (a).

[0055] Fig. 6A An exemplary process 602 for building an AR+ Internet using architecture 500 according to some embodiments is shown. Process 602 can be applied to various application usage scenarios in 604, such as education, healthcare, transportation, virtual shopping, entertainment, games, etc. For the content creation portion 606 of process 602, a site builder application and a hosting cloud system (e.g., cloud system 210) can be used for the creation and storage of AR content. For the content discovery portion 608 of process 602, a DNS+ system (e.g., an on-chain DNS system 512) can be used to retrieve URLs corresponding to AR content. A search engine can be used to retrieve URLs corresponding to AR content from the DNS+ system. For the content consumption portion 610, an AR+ browser 612 can be run on a user terminal device to render and display AR content. Display scene 614 shows an exemplary display of an AR+ browser in combination with physical objects in the physical world.

[0056] Embodiment AR+Internet architecture 500 supports interoperability and accessibility. The architecture supports platform-independent cloud and edge computing and naturalized human-computer interfaces. Embodiment AR+Internet architecture 500 also supports scalability. The architecture supports decentralized systems that enable cloud and edge computing, and includes a blockchain-based content addressable Internet. Embodiment AR+Internet architecture 500 is also robust. The architecture includes an AR+ browser that naturally supports active content presentation. The architecture supports a decentralized ecosystem.

[0057] Figure 6BAn exemplary system for building an AR+ Internet using architecture 500 according to some embodiments is shown. For content creation 606, existing modeling and analysis techniques known in the art can be used to create AR content. The created AR content can be stored in a distributed cloud storage system (e.g., cloud system 210). A search engine can be used to search DNS+ (e.g., on-chain DNS system 512) to find the URL corresponding to the AR content. For content consumption 610, a user terminal device (e.g., a mobile phone) can query DNS+ to obtain the URL corresponding to the AR content and use the URL to download the AR content, as shown in reference Figure 2 The AR+ browser application can render and display the downloaded AR content. The downloaded AR content can be displayed on the display screen of the user terminal device. The AR content can also be displayed on another AR display device (such as an AR headset, AR glasses, etc.) connected to the user terminal device. Figure 6B In the system shown, on a decentralized AR+ website, related AR content can be securely stored on the blockchain's distributed ledger, controlled by the owner using his private key. These files can be mapped to a human-readable blockchain domain. When a user terminal device sends a ping command to the domain, the user terminal device can access the associated website.

[0058] Fig. 7A An embodiment of the AR+ Internet Domain Name Service according to some embodiments, the Space Object Name Service (SONS), is shown. The Domain Name System (DNS) (e.g., system 702) was established in 1983 and is a public directory of domain names and IP addresses of resources hosted on the web. DNS is managed by the Internet Corporation for Assigned Names and Numbers (ICANN), which makes it a central point of failure and control on the Internet. SONS (e.g., system 704) has some similarities to traditional DNS, but includes more improvements. SONS 704 makes full use of and uses blockchain. SONS can map individual blockchain addresses to human-readable domain names. SONS domains can use globally distributed computer networks, such as P2P networks 706, instead of pointing to servers located in a central location. SONS domains can be represented as tokenized assets on the blockchain. SONS domains can be mapped to other addresses pointed to by user terminal devices on the network according to requests from user terminal devices. On the AR+ website, relevant content (e.g., URLs) can be securely stored on the distributed ledger of the blockchain, controlled by the owner using his private key. SONS domains can be mapped to human-readable blockchain domains. When a user pings that domain, they can visit the associated website.

[0059] Figure 7BExemplary SONS spatial object identifier (SOid) and SONSLink generation and retrieval according to some embodiments are shown. After creating AR content, the space object (SO) feature generator 712 can map the AR content to generate a unique feature vector called a feature ID (FID). Feature IDs can be generated in a unimodal format or a multimodal format through images, videos, and any other type of media. The feature ID can be a unique mapping of a unique known physical identifier, such as the geographic location of a permanent object, a QR code of static content or an object, etc. The feature ID can also be a unique identifier for a combination of trunk features, neck features, and head features discussed in this disclosure. This flexibility enables the generation of robust unique feature IDs for different types of spatial objects (e.g., physical objects in the physical world), whether the spatial object is permanent or temporary, static or dynamic, mobile or stable.

[0060] Next, the SOid generator 714 can split the FID into smaller chunks, each of which is cryptographically hashed. Smaller chunks from the FID can be given a unique fingerprint, called a SOid (e.g., p4wX6UsktY2jvbje5r6CAjNpwCxlt72N). The SOid serves as a permanent record of the AR content. The SONLink generator 725 can map the SOid to a human-readable SONSLink address (e.g., SONLink / foo.example.sod / directory / image / category1 / sample1.jpg). The SONSlink address (e.g., a URL discussed in this disclosure, such as the URL discussed with respect to operation 216) can be registered and stored in the blockchain SONS database 728 together with the SOid.

[0061] Figure 7C SONS storage and retrieval based on an embodiment blockchain according to some embodiments are shown. The creator of AR content can upload the AR content to the storage layer 712. For example, the AR content can be uploaded to IPFS (e.g., cloud system 210). Information about the AR content (such as SOid and SONSLink) can be registered with a distributed SONS database. The distributed SONS database can create a new block in the blockchain in the blockchain network layer 724 to store information about the AR content (e.g., SOid and SONSLink) in the new block created in the blockchain. On the consumer user side, the consumer can obtain information about the AR content (e.g., URL) and use the information to download the AR content from IPFS (e.g., cloud system 210) after verifying the information in the blockchain layer 724.

[0062] Fig.7D An exemplary SONS hierarchy 730 is shown in accordance with some embodiments. The SONS hierarchy 730 includes a top level 732, a second level 734, and at least one child level 736. Fig.7D Three levels in the SONS hierarchy 730 are shown, but the number of levels is not limited to three. The SONS hierarchy 730 may also include two levels (eg, no children) or more than three levels (eg, multiple children).

[0063] Fig. 7E 74 shows an exemplary system in the AR+Internet according to some embodiments. When an AR+Internet consumer uses an AR+web browser application running on a user terminal device 742, the user terminal device can perform feature extraction on a physical object (e.g., Figure 2 ), the user terminal device 742 may send a SONS query request to the internal SONS server 744. The SONS query request may include matching features of the physical object. The internal SONS server 744 and the user terminal device 742 may be located on the same side of the firewall 746. The internal SONS server 744 may forward the SONS query request to the external SONS server 748 through the firewall 746. The external SONS server 748 may then forward the SONS query request to SONS 750. SONS 750 may use an AR+ parser to map the features of the physical object in the SONS query request to a feature ID. The feature ID is then converted to a SOid through a SOid generator, as shown in reference Figure 7C The search engine in SONS 750 can use the SOid to find the block in the blockchain 752 where the SOid is stored. After verifying the block, the SONSLink address (e.g., URL) stored in the same block is returned to the user terminal device 742 in the SONS query response. Then, using, for example, Figure 2 218 to 228, the AR+ browser may use the SONSLink address in the SONS query response to obtain information from the cloud system 754 (e.g., Figure 5 AR content in the chain cloud 514 or Figure 7C The AR content corresponding to the physical object is retrieved from IPFS in .

[0064] Figure 8A flowchart of a method 800 for retrieving AR content corresponding to a physical object using SONS according to some embodiments is shown. Method 800 may be performed or carried out by an AR+ browser executed on one or more processing units of a computing device (e.g., a user terminal device 204). Examples of processing units include, but are not limited to, a graphics processing unit (GPU), a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an artificial intelligence (AI) accelerator, or a combination thereof. Method 800 may also be performed or carried out by a routine, a subroutine, or a software module executed by one or more processing units. Method 800 may also be performed or carried out by a combination of hardware and software. It will be appreciated by those of ordinary skill in the art that the present disclosure may decode software for implementing or executing method 800. Method 800 may include more or fewer operations than those shown and described, and may be performed or carried out in a different order. Computer readable code or instructions of software executable by one or more processing units may be stored in a non-transitory computer readable medium, such as a memory of a computing device.

[0065] Method 800 begins at operation 802, where a computing device performs objection detection on the media content of a physical object to match the characteristics of the physical object. In operation 804, the computing device sends a SONS query request to a spatial object name service (SONS) server. The SONS query request includes the characteristics of the physical object. In operation 806, the computing device receives a SONS query response from the SONS server. The SONS query response includes a domain name system (DNS) domain name for retrieving augmented reality (AR) content corresponding to the physical object, and the DNS domain name is based on the characteristics of the physical object. In operation 808, the computing device obtains an Internet Protocol (IP) address by resolving the DNS domain name. In operation 810, the computing device obtains the AR content corresponding to the physical object based on the IP address and the SONS query response. In operation 812, the computing device renders at least a portion of the AR content corresponding to the physical object. A portion of the AR content may correspond to a first portion of the physical object. A portion of the AR content may be mixed (e.g., visually mixed) with a second portion of the physical object.

[0066] In some embodiments, the media content of the physical object may include at least one of one or more images of the physical object, audio of the physical object, or text of the physical object.

[0067] In some embodiments, performing objection detection may include: inputting media content into a trunk classification engine to obtain trunk features from the trunk classification engine; inputting trunk features into a neck classification engine to obtain neck features from the neck classification engine; and inputting neck features into a head classification engine to obtain head features from the head classification engine.

[0068] In some embodiments, the trunk classification engine may include a trunk neural network. The neck classification engine may include a neck neural network. The head classification engine may include a head classification neural network. In some embodiments, the features of the physical object in the SONS query request sent to the SONS server may include trunk features, neck features, and head features. In some embodiments, the SONS server may generate a uniform resource locator (URL) based on a combination of trunk features, neck features, and head features. The head features may include mapping information from media content to semantic information of the physical object, and the semantic information is associated with the URL. In some embodiments, the mapping information may be stored in a block of the blockchain.

[0069] In some embodiments, the features of the physical object may correspond to the body of the physical object.

[0070] Fig. 9 A block diagram of a computing system 900 that can be used to implement the devices and methods disclosed herein according to some embodiments is shown. For example, the computing system can be a user terminal device, a device running in a distributed cloud system, a device running in a SONS, or any entity in any computing device described above. A particular device may use all of the components shown or only a subset of the components, and the level of integration may vary from device to device. In addition, a device may include multiple instances of a component, such as multiple processing units, processors, memories, transmitters, receivers, and the like. The computing system 900 includes a processing unit 902. The processing unit includes a central processing unit (CPU) 914, a memory 908, and may also include a mass storage device 904, a video adapter 910, and an I / O interface 912 connected to a bus 920.

[0071] Bus 920 can be one or more of several bus architectures of any type, including a storage bus or storage controller, a peripheral bus, or a video bus. CPU 914 can include any type of electronic data processor. Memory 908 can include any type of non-transient system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM) or a combination thereof. In one embodiment, memory 908 can include a ROM used at startup and a DRAM for storing programs and data, which is used when executing a program.

[0072] The mass storage 904 may include any type of non-transitory storage device for storing data, programs, and other information and making the data, programs, and other information accessible via the bus 920. For example, the mass storage 904 may include one or more of a solid-state drive, a hard disk drive, a magnetic disk drive, or an optical disk drive.

[0073] The video adapter 910 and the I / O interface 912 provide interfaces for coupling external input and output devices to the processing unit 902. As shown, examples of input and output devices include a display 918 coupled to the video adapter 910 and a mouse, keyboard, or printer 916 coupled to the I / O interface 912. Other devices may be coupled to the processing unit 902, and more or fewer interface cards may be used. For example, a serial interface such as a universal serial bus (USB) (not shown) may be used to provide an interface for external devices.

[0074] The processing unit 902 also includes one or more network interfaces 906, which may include wired links such as Ethernet cables to access nodes or different networks, or wireless links. The network interface 906 supports the processing unit 902 to communicate with remote units through the network. For example, the network interface 906 can provide wireless communication through one or more transmitters / transmit antennas and one or more receivers / receive antennas. In one embodiment, the processing unit 902 is coupled to a local area network 922 or a wide area network for data processing and communication with remote devices (e.g., other processing units, the Internet, or remote storage facilities).

[0075] It should be understood that one or more steps in the embodiment methods provided herein can be performed by corresponding units or modules. The corresponding units or modules can be hardware, software or a combination thereof. For example, one or more of these units or modules can be integrated circuits, such as field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs).

[0076] Although the specification has been described in detail, it should be understood that various changes, substitutions and modifications may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. In addition, the scope of the present disclosure is not intended to be limited to the specific embodiments described herein, and as one of ordinary skill in the art will readily appreciate from this disclosure, processes, machines, manufactures, compositions of matter, components, methods or steps (currently existing or to be developed later) may perform substantially the same functions or obtain substantially the same results as the corresponding embodiments described herein. Therefore, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, components, methods or steps.

Claims

1. A method, characterized in that include: The computing device performs objection detection on the media content of the physical object to match the characteristics of the physical object; The computing device sends a spatial object name service (SONS) query request to a SONS server, wherein the SONS query request includes the feature of the physical object; The computing device receives a SONS query response from the SONS server, the SONS query response comprising a Domain Name System (DNS) domain name for retrieving augmented reality (AR) content corresponding to the physical object, the DNS domain name being based on the characteristic of the physical object; The computing device obtains an Internet Protocol (IP) address by resolving the DNS domain name; Retrieving, by the computing device, the AR content corresponding to the physical object based on the IP address and the SONS query response; The computing device renders at least a portion of the AR content corresponding to the physical object, the portion of the AR content corresponding to a first portion of the physical object, and the portion of the AR content is to be blended with a second portion of the physical object.

2. The method according to claim 1, characterized in that The media content of the physical object includes at least one of one or more images of the physical object, audio of the physical object, or text of the physical object.

3. The method according to claim 1, characterized in that The execution objection detection includes: The computing device inputs the media content into a backbone classification engine to obtain backbone features from the backbone classification engine; The computing device inputs the trunk feature into a neck classification engine to obtain a neck feature from the neck classification engine; The computing device inputs the neck features into a head classification engine to obtain head features from the head classification engine.

4. The method according to claim 3, characterized in that: The trunk classification engine includes a trunk neural network, the neck classification engine includes a neck neural network, and the head classification engine includes a head classification neural network.

5. The method according to claim 3, characterized in that: The features of the physical object in the SONS query request sent to the SONS server include the trunk features, the neck features, and the head features.

6. The method according to claim 3, characterized in that: The SONS server locates the DNS domain name based on a combination of the trunk feature, the neck feature, and the head feature.

7. The method according to claim 6, characterized in that The SONS server generates a uniform resource locator (URL) based on the combination of the trunk feature, the neck feature, and the head feature, and wherein the head feature includes mapping information from the media content to semantic information of the physical object, the semantic information being associated with the URL.

8. The method according to claim 7, characterized in that The mapping information is stored in the blocks of the blockchain.

9. The method according to any one of claims 1 to 8, characterized in that The feature of the physical object corresponds to a body of the physical object.

10. A device, characterized in that: include: at least one processor; A non-transitory computer-readable storage medium storing a program, the program comprising instructions that, when executed by the at least one processor, cause the apparatus to: performing objection detection on media content of a physical object to match characteristics of the physical object; Sending a spatial object name service (SONS) query request to a SONS server, wherein the SONS query request includes the feature of the physical object; receiving a SONS query response from the SONS server, the SONS query response comprising a Domain Name System (DNS) domain name for retrieving augmented reality (AR) content corresponding to the physical object, the DNS domain name being based on the characteristic of the physical object; Obtaining an Internet Protocol (IP) address by resolving the DNS domain name; retrieving the AR content corresponding to the physical object based on the IP address and the SONS query response; At least a portion of the AR content is rendered corresponding to the physical object, the portion of the AR content corresponding to a first portion of the physical object and the portion of the AR content is to be blended with a second portion of the physical object.

11. The device according to claim 10, characterized in that The media content of the physical object includes at least one of one or more images of the physical object, audio of the physical object, or text of the physical object.

12. The device according to claim 10, characterized in that The instructions for causing the apparatus to perform the objection detection include instructions for causing the apparatus to: Inputting the media content into a backbone classification engine to obtain backbone features from the backbone classification engine; Inputting the backbone features into a neck classification engine to obtain neck features from the neck classification engine; The neck features are input into a head classification engine to obtain head features from the head classification engine.

13. The device according to claim 12, characterized in that The trunk classification engine includes a trunk neural network, the neck classification engine includes a neck neural network, and the head classification engine includes a head classification neural network.

14. The device according to claim 12, characterized in that The features of the physical object in the SONS query request sent to the SONS server include the trunk features, the neck features, and the head features.

15. The device according to claim 12, characterized in that The SONS server locates the DNS domain name based on a combination of the trunk feature, the neck feature, and the head feature.

16. The device according to claim 15, characterized in that The SONS server generates a uniform resource locator (URL) based on the combination of the trunk feature, the neck feature, and the head feature, and wherein the header feature includes mapping information from the media content to semantic information of the physical object, the semantic information being associated with the URI.

17. The device according to claim 16, characterized in that The mapping information is stored in the blocks of the blockchain.

18. The device according to any one of claims 10 to 17, characterized in that The feature of the physical object corresponds to a body of the physical object.

19. A non-transitory computer readable medium, characterized in that Instructions are stored therein, and when the instructions are executed by the device, the device performs operations, including: performing objection detection on media content of a physical object to match characteristics of the physical object; Sending a space object name service (SONS) query request to a SONS server, wherein the SONS query request includes the feature of the physical object; receiving a SONS query response from the SONS server, the SONS query response comprising a Domain Name System (DNS) domain name for retrieving augmented reality (AR) content corresponding to the physical object, the DNS domain name being based on the characteristic of the physical object; Obtaining an Internet Protocol (IP) address by resolving the DNS domain name; retrieving the AR content corresponding to the physical object based on the IP address and the SONS query response; At least a portion of the AR content is rendered corresponding to the physical object, the portion of the AR content corresponding to a first portion of the physical object, and the portion of the AR content is to be blended with a second portion of the physical object.

20. The non-transitory computer readable medium of claim 19, wherein: The media content of the physical object includes at least one of one or more images of the physical object, audio of the physical object, or text of the physical object.

Images