System for dynamic augmented reality

Abstract

A system for dynamically augmented reality is provided. The system includes one or more processors, and a non-transitory computer-readable storage medium that includes instructions that, when executed by the one or more processors, cause the one or more processors to perform operations that include: at a backend server, at least one of the backend server and the one or more processors; detecting activation of the deep link from a third-party resource executing at the device, the back-end server providing a message client implemented by one or more processors of the device, the third-party resource being provided by an entity different from a provider of the back-end server; obtaining respective values for one or more launch attributes of the augmented reality component from the payload of the deep link; and causing a message client to be launched at the device and loading the augmented reality component in the message client.

Description

[0001] This invention application is a divisional application of invention patent application No. 202180041688.6, with the application date of June 3, 2021, international application number PCT / US2021 / 035630, and entered the Chinese national stage on December 9, 2022, and the invention name is "Methods, systems and machine-readable non-transitory storage media for dynamic augmented reality". Technical Field

[0002] The present disclosure generally relates to facilitating interaction between messaging clients and third-party resources. Background Art

[0003] The popularity of computer-implemented programs that allow users to access content online and interact with other users continues to grow. There are various computer-implemented applications that allow users to share content with other users through a messaging client. Some of these computer-implemented applications (referred to as apps) can be designed to run on mobile devices such as phones, tablets, or watches, while having back-end services provided on a server computer system to perform operations that may require more resources than reasonable resources to perform at the client device (e.g., storing large amounts of data or performing computationally expensive processing). For example, a messaging client and an associated messaging server system can be configured to allow online users to share content. Summary of the Invention

[0004] In one aspect, the present invention provides a system for dynamic augmented reality. The system includes one or more processors and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: detecting, at a backend server, activation of a deep link from a third-party resource executed at a device, the backend server providing a messaging client implemented by one or more processors of the device, the third-party resource provided by an entity different from the provider of the backend server; obtaining corresponding values ​​of one or more launch properties for an augmented reality component from a payload of the deep link; and causing the messaging client to be launched at the device and the augmented reality component to be loaded into the messaging client.

[0005] In another aspect, the present invention provides a system comprising: one or more processors; and a non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting, at a server, a request from a third-party resource executed at a user system, the third-party resource being provided by an entity different from a provider of the server; obtaining corresponding values ​​for one or more properties of an augmented reality component from the request; causing, at the user system, the augmented reality component to be loaded into an interactive application executed at the user system; and causing, at the user system, the interactive application to display content captured by a camera device of the user system that is modified by the augmented reality component according to the values ​​of the one or more properties.

[0006] In another aspect, the present invention provides a system comprising: one or more processors; and a non-transitory computer-readable storage medium comprising instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting, at a back-end server, a request from a third-party resource executed at a user system, the back-end server providing an interactive application implemented by one or more processors of the user system, the third-party resource being provided by an entity different from a provider of the back-end server; obtaining corresponding values ​​of one or more startup properties for an augmented reality component from the request; and causing the interactive application to be started at the user system and the augmented reality component to be loaded into the interactive application. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numerals refer to the figure in which the element is first introduced. Some embodiments are shown by way of example and not limitation in the accompanying drawings, in which:

[0008] Figure 1 is a diagrammatic representation of a networked environment in which a developer tool system may be deployed, according to some examples.

[0009] Figure 2 is a block diagram of the architecture of a system for providing dynamic augmented reality components according to some examples.

[0010] Figure 3 is a flowchart of a method for providing a dynamic augmented reality component according to some examples.

[0011] Figure 4 is a diagrammatic representation of example content captured from a camera view screen of a messaging client.

[0012] Figure 5 is a diagrammatic representation of example content captured from a camera view of a messaging client and modified using a dynamic augmented reality component, according to some examples.

[0013] Figure 6 is a diagrammatic representation of a machine in the form of a computer system, according to some examples, within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0014] The messaging server system (which hosts the backend services of the associated messaging client) is configured to allow a user to capture images and videos using a camera provided to a client device (which hosts the messaging client) and to share the captured content with other users via network communications. The messaging server system is also configured to provide an augmented reality (AR) component accessible via the messaging client that can be used to modify the content captured by the camera, for example, by overlaying graphics or animations on top of captured images or video frames and by adding three-dimensional (3D) effects, objects, characters, and transformations.

[0015] The AR component can be implemented using a programming language suitable for app development (such as JavaScript or Java). The AR component can be executed within the messaging client when the camera device records the video, so that each captured frame is modified in a manner determined by the specific AR component configuration. For example, the AR component can be configured to detect the head position of the person being captured by the video in each frame of the video, and superimpose the image of a party hat on the detected head position on each frame, so that viewers of the video captured using the AR component will see that the person being captured in the video is wearing a party hat. When the camera view screen is opened in the messaging client, the effects of the AR component can be applied to the captured image, and the messaging client can present one or more visual controls to the user, which allow the application and removal of visual effects generated by the AR component, as well as the selection of the desired AR component from multiple AR components.

[0016] A user can launch an AR component by opening the messaging client (in some embodiments, opening it to a camera view that is preparing to capture or has already captured an image or video) and selecting the icon representing the desired AR component. AR components are identified in the messaging server system by their corresponding AR component identifiers. In some embodiments, a specific AR component maintained by the messaging server system can be accessed directly from a third-party resource, such as a third-party app, via a deep link that references the AR component identifier, without first having to launch the messaging client (whether a standalone app or a web-based version of the app). In the context of an app, a deep link can be described as using a Uniform Resource Identifier (URI) that links to a specific location within the app, rather than simply launching the app. When such a deep link is activated from a third-party app executed on a camera-equipped client device, the deep link is passed to the messaging server system and returned to the client device's live camera, where the AR component referenced by the AR component identifier in the deep link is activated. For the purposes of this description, activating an AR component in the messaging client renders the AR component available for selection by the user. Launching the AR component causes the AR component's effects to be applied to the captured content. Activating the live camera and launching the given AR component together causes the effects of the given AR component to be applied to the content being captured by the camera. In addition to providing access to the AR component directly from third-party apps, the message server system can be configured to allow third-party developers to share content originating from their third-party apps to the AR component maintained by the message server system.

[0017] The technical problem of adding content originating from a third-party app to an AR component maintained by a message server system is solved by configuring the AR component to modify content captured by a camera device based on the value of one or more launch properties, and allowing the values ​​to be delivered from the third-party app to the message server system in the payload of a deep link that references the AR component's AR component identifier. The content provided to the AR component maintained by the message server system that originates from the third-party app (e.g., sports scores provided by a sports news channel app or current weather information provided by a weather app) may be dynamic in that the content may change over time.

[0018] For the purposes of this description, an AR component that is configured to modify the content captured by the camera according to the value of one or more launch attributes is referred to as a dynamic augmented reality component or dynamic AR component. The modification type specified by the launch attribute can indicate that a certain type of content (e.g., text, image, or animation) will be superimposed on a portion of the captured image (e.g., centered on the lower right quadrant of the frame area). The value of such a launch attribute can be, for example, text, an image, or an animation. For example, the value of a launch attribute provided from a sports news channel app can be an image or string indicating the cumulative score of an ongoing sporting event. For the purposes of this description, the value of a launch attribute that can be delivered to a message server system via a deep link can be simply referred to as an attribute. Another example of a launch attribute is a launch attribute that references a binary asset (e.g., a three-dimensional machine learning model that takes content captured by a camera as input), which can be executed relative to the captured content, resulting in the entire object in the captured frame (e.g., a human body depicted in the frame) being replaced with a three-dimensional (3D) mesh or texture.

[0019] The messaging server system is configured to provide users with a set of developer tools, including features that allow third-party app developers to include deep links in their third-party apps that reference an AR component identifier and include values ​​for one or more launch properties in the payload. The developer tools provided by the messaging server system can be invoked from a third-party computer system via a developer portal, which can be accessed via a web browser. The developer portal can be downloaded to the third-party computer system, in which case a web browser may not be required. The developer portal can be used to obtain deep links that reference a given AR component identifier and access an AR component service application programming interface (API), which is configured to allow third-party developers to create new dynamic AR components. In some embodiments, the messaging server system pushes an AR component template and properties and assets that determine the functionality of the new AR component to the developer portal via the AR component service API. A user interface (UI) provided with the developer portal allows third-party developers to select any properties for the new AR component and specify at least one property whose value will be received from the third-party app via the deep link.

[0020] In operation, a third-party app uses developer tools provided by the messaging server system to construct a request to link to an AR component with a specific AR component identifier. In some examples, the third-party app is a registered client with respect to the developer tools provided by the messaging server system. The developer tools system constructs the deep link and encrypts the payload, which stores the AR component identifier and properties ready for launch.

[0021] When the messaging server system detects activation of a deep link from a client device (e.g., when a user activates the deep link using associated visual controls provided by a third-party app's UI), the messaging server system decrypts the deep link payload and obtains the AR component identifier, verifies that the AR component referenced by the AR component identifier in the deep link is valid, and verifies that the AR component is unlockable (meaning it can be activated and immediately used in the messaging app). Properties are then extracted from the deep link's payload. The extracted properties are stored in a designated data object associated with the AR component identifier at a persistent storage system. This data object is configured, for example, by a developer using a scripting language via the developer portal 132 to be populated with the properties from the deep link's payload when the associated AR component is initialized. When the AR component is launched, the stored properties are exposed to the AR component using this data object. The persistent storage system can be associated with the messaging server system, or it can reside at the client computer system hosting the messaging client.

[0022] Once the AR component is activated, the content captured from the camera view is enhanced with the modifications provided by the AR component. In some embodiments, sending or publishing the captured content enhanced with the modifications provided by the AR component enables the messaging client to have the AR component available to the user for a predetermined period of time (e.g., 48 hours). The properties extracted from the payload of the deep link and stored in the persistent storage system remain in the persistent storage system until the next time the deep link is enabled from the third-party app, at which time these stored properties are overwritten by new properties, which are the properties extracted from the link at that time. In the case where the properties extracted from the deep link are references to binary assets, such properties indicate the storage location of the referenced binary asset, which may be at the client computer device hosting the messaging client.

[0023] The methods described herein can be beneficially used for various third-party resources. Third-party resources can be understood as installed apps, as well as HTML-based mini-apps and games. HTML is a markup language for structuring and presenting content on the World Wide Web. Installed apps are software files that are installed on a device using executable code and can be launched independently of a messaging client. Mini-apps and games (which can be thought of as tiny apps that can be downloaded to a client device and run within a large app) are HTML-based apps that require the messaging client to download a markup language document and present the markup language document in a browser (in a web view) run by the messaging client. When launched by the user, the HTML file is retrieved from the server for presenting the app.

[0024] Figure 11 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each of which hosts several applications including a messaging client 104 and a third-party app 103. The third-party app 103 is configured to allow users to access functionality provided by a third-party system 130. Each messaging client 104 is communicatively coupled to a messaging server system 108 and other instances of the messaging client 104 via a network 106 (e.g., the Internet).

[0025] The message client 104 is capable of communicating and exchanging data with another message client 104 and a message server system 108 via the network 106. The data exchanged between the message clients 104 and between the message clients 104 and the message server system 108 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0026] The message server system 108 is capable of providing server-side functionality to specific message clients 104 via the network 106. Although certain functions of the message system 100 are described herein as being performed by the message client 104 or by the message server system 108, the location of certain functions within the message client 104 or within the message server system 108 may be a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the message server system 108 but later migrate the technologies and functions to a message client application 104 on a client device 102 that has sufficient processing power.

[0027] The message server system 108 supports various services and operations provided to the message clients 104. Such operations include sending data to the message clients 104, receiving data from the message clients 104, and processing data generated by the message clients 104. As examples, the data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the message system 108 is activated and controlled by functions available through the user interface (UI) of the message client 104.

[0028] Turning now specifically to the message server system 108, an application program interface (API) server 110 is coupled to an application server 112 and provides a programming interface to the application server 112. The application server 112 is communicatively coupled to a database server 118 that facilitates access to a database 120. A web server 124 is coupled to the application server 112 and provides a web-based interface to the application server 112. To this end, the web server 124 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols. The database 120 stores data associated with the messages processed by the application server 112, such as profile data about a particular entity. In the case where the entity is a person, the profile data includes, for example, the user's name, notification and privacy settings, the user's self-reported age, and records related to changes made by the user to their profile data.

[0029] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the message client 104 can call or query to activate the functions of the application server 112. The API server 110 exposes various functions supported by the application server 112, including: account registration; login functionality; sending messages from a particular message client 104 to another message client 104 via the application server 112; sending media files (e.g., images or videos) from a message client 104 to the message server system 114 for possible access by another message client 104; opening application events (e.g., related to the message client 104); and various functions supported by developer tools provided by the message server system 108 for use by third-party computer systems.

[0030] The application server 112 hosts several server applications and subsystems, including, for example, a message server system 114, an image processing server 116, and a social network server 122. The message server system 114 implements several message processing technologies and functions, particularly those related to aggregating and otherwise processing content (e.g., text and multimedia content) included in messages received from multiple instances of the message client 104. The image processing server 116 is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the message server system 114. The social network server 122 supports various social networking functions and services and makes these functions and services available to the message server system 114.

[0031] Figure 1Also shown is a developer tool server 117. Developer tool server 117 maintains one or more software developer kits (SDKs) that allow users to integrate features provided by the messaging server system across their apps (also referred to as third-party apps). These features include deep linking of dynamic AR components that are configured to modify content captured by a camera based on the values ​​of one or more launch properties delivered from the third-party app to the messaging server system in the payload of the deep link, which references the AR component's AR component identifier. The functionality provided by developer tool server 117 can be accessed from a third-party computer system via a developer portal accessible via a web browser. In some examples, the developer portal that provides access to the functionality provided by developer tool server 117 to a third-party computer system (e.g., third-party system 130) can be downloaded to the third-party computer system, in which case a web browser may not be required. Third-party system 130 is shown as including developer portal 132. As described above, developer portal 132 can be accessed via a web browser executed on third-party app provider system 130 or downloaded to third-party computer system 130.

[0032] Figure 2 is a block diagram illustrating an example system 200 for providing a dynamic AR component. The system 200 includes a deep link generator 210, a deep link activation detector 220, an attribute extractor 230, and an augmented reality component activator 240. As mentioned above, for the purposes of this description, the augmented reality component may be referred to as an AR component. The deep link generator 210 is configured to construct a deep link that includes a method for storing a representation of the deep link. Figure 1 The message server system 108 includes an identifier of the AR component (augmented reality component identifier) ​​and a payload for storing the value of one or more activation attributes. As explained above, the AR component is configured to modify the content captured by the camera device according to the value of one or more activation attributes. The deep link activation detector 220 is configured to detect the activation of a link from a third-party resource (e.g., from a client device) executed at the client device. Figure 1 The third-party app 103 executed on the client device 102 activates the deep link. The attribute extractor 230 is configured to extract corresponding values ​​of one or more launch attributes from the payload of the deep link and store the extracted corresponding values ​​in the persistent storage system.

[0033] The augmented reality component activator 240 is configured to cause the messaging client 104 to launch into a camera view screen. The camera view screen indicates that the camera is activated. The augmented reality component activator 240 dynamically configures the AR component using the corresponding values ​​when the corresponding values ​​become available or are updated in the persistent storage system. After the AR component is dynamically configured, the AR component is ready to modify the captured content according to the stored corresponding values. The augmented reality component activator 240 then loads the dynamically configured AR component into the messaging client. Each of the various components of the system 200 can be set Figure 1 1 and / or the message server system 108. Additional details regarding the operation of the system 200 are provided below.

[0034] Figure 3 is a flow chart of a method 300 for providing a dynamic AR component. The method 300 may be performed by processing logic that may include hardware (e.g., dedicated logic, programmable logic, microcode, etc.), software, or a combination thereof. In one example embodiment, some or all of the processing logic resides in Figure 1 The client device 102 and / or Figure 1 At the message server system 108 of the client device. At operation 310, the message server system 108 detects a request from a third party resource (e.g., from a Figure 1 Activation of a deep link by a third-party app 103 executed at a client device 102 of the messaging server system 108. The deep link is constructed to include in its payload an AR component identifier representing the AR component in the messaging server system 108 and values ​​of one or more launch attributes. At operation 320, the values ​​of the one or more launch attributes are extracted and stored in a persistent storage system. As mentioned above, the persistent storage system can reside at the client device 103 or the messaging server system 108. At operation 330, the messaging client 104 is caused to launch to a camera view screen. At operation 340, the corresponding values ​​are retrieved from the persistent storage system and the corresponding values ​​are used to configure the AR component so that it can be used to modify the content captured by the camera according to the stored corresponding values. At operation 350, the configured AR component is loaded into the messaging client.

[0035] Although the described flowchart may show the operations as a sequential process, many of the operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. When its operations are completed, the process terminates. A process can correspond to a method, a procedure, an algorithm, etc. The operations of a method can be performed in whole or in part, can be performed in combination with some or all of the operations in other methods, and can be performed by any number of different systems (such as the systems described herein) or any part thereof (such as a processor included in any system).

[0036] Figure 4 4 is a diagrammatic representation of example content captured from a camera view of a messaging client 104. A person is depicted in region 410 as being captured in a still image or video frame. Visual controls 420 can be activated to apply modifications to the associated AR component to the captured content. In this example, visual controls 420 can be executed to enhance the captured content with current sports scores obtained from a third-party app 103, which the messaging client 104 launched from via a deep link. Figure 5 is a diagrammatic representation 500 of content captured from a camera view of a messaging client, modified using an AR component referenced in a deep link and based on a value extracted from the deep link's payload (in this example, the current sports score). Shown in area 510 are: a person captured in a still image or video frame; and also shown in area 530 as the current sports score superimposed on the captured content. Visual controls 520 are shown using a thicker black line to indicate that the associated AR component is activated. A deep link is Figure 1 The process of constructing a deep link includes additional variables in the payload of the deep link. The following is an example 1 showing the process of generating a deep link in the payload of the deep link. Figure 5 Example attributes of the current sports score are shown in region 530 of the diagram - the variables team_one_score, team_two_score, team_one_name, team_two_name and their associated values.

[0037] Example 1

[0038] team_one_score:5, team_two_score:3, team_one_name:

[0039] "Team Beagle", team_two_name: "Team Beaver"

[0040] The properties shown in Example 1 are stored in a data object associated with the AR component identifier referenced in the deep link. When the AR component is loaded into the messaging client 104, these properties are passed directly from the data object to the AR component. In one embodiment, using Figure 5 In the example shown in , the AR component can be configured to modify binary assets within the AR component. For example, the AR component can be configured to highlight the team name and score using the team colors and can also be configured to display the associated team logo. The team colors and logo image can be predefined in the AR component and selected based on properties provided in the deep link payload.

[0041] Machine Architecture

[0042] Figure 6 6 is a diagrammatic representation of a machine 600 in which instructions 608 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed for causing the machine 700 to perform any one or more of the methodologies discussed herein. For example, the instructions 608 may cause the machine 600 to perform any one or more of the methodologies described herein. The instructions 608 convert a general-purpose, non-programmed machine 600 into a specialized machine 600 that is programmed to perform the functions described and illustrated in the manner described. The machine 600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 600 may operate as a server or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing instructions 608 specifying actions to be taken by the machine 600, either sequentially or otherwise. Furthermore, while only a single machine 600 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 1100 to perform any one or more of the methods discussed herein. For example, the machine 600 may include a client device 102 or any of several server devices forming part of the message server system 108. In some examples, the machine 600 may also include both a client and server system, where certain operations of a particular method or algorithm are performed on the server side and certain operations of a particular method or algorithm are performed on the client side.

[0043] The machine 600 may include a processor 602, a memory 604, and input / output (I / O) components 638 that may be configured to communicate with each other via a bus 640. In an example, the processor 602 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 606 that executes instructions 608 and a processor 610. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions concurrently. Although Figure 6 Multiple processors 602 are shown, but the machine 600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0044] The memory 604 includes a main memory 612, a static memory 614, and a storage unit 616 that are accessible to the processor 602 via the bus 640. The main memory 604, the static memory 614, and the storage unit 616 store instructions 608 that implement any one or more of the methodologies or functions described herein. The instructions 608 may also reside, completely or partially, within the main memory 612, within the static memory 614, within the machine-readable medium 618 within the storage unit 616, within at least one of the processors 602 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 600.

[0045] The I / O components 638 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 638 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It will be understood that the I / O components 638 may include Figure 6Many other components are not shown in the drawings. In various examples, the I / O components 638 may include user output components 624 and user input components 626. The user output components 624 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The user input components 626 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0046] In other examples, the I / O component 638 may include a biometric component 628, a motion component 630, an environmental component 632, or a positioning component 634, as well as various other components. For example, the biometric component 628 includes components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 630 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0047] Environmental components 632 include, for example, one or more cameras (with still image / photo and video capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0048] With respect to cameras, client device 102 can have a camera system including, for example, a front-facing camera on the front surface of client device 102 and a rear-facing camera on the rear surface of client device 102. The front-facing camera can, for example, be used to capture still images and videos of a user of client device 102 (e.g., a "selfie"), which can then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera can be used to capture still images and videos in a more traditional camera mode, which are similarly enhanced using the enhancement data. In addition to the front and rear-facing cameras, client device 102 can also include a 360° camera for capturing 360° photos and videos.

[0049] Additionally, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the client device 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0050] The location component 634 includes a positioning sensor component (eg, a GPS receiver component), an altitude sensor component (eg, an altimeter or a barometer that detects air pressure from which altitude can be obtained), an orientation sensor component (eg, a magnetometer), and the like.

[0051] A variety of technologies can be used to implement communications. The I / O components 638 also include a communication component 636 that is operable to couple the machine 600 to the network 620 or device 622 via corresponding couplings or connections. For example, the communication component 636 may include a network interface component or another suitable device to interface with the network 620. In other examples, the communication component 636 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low power consumption), Device 622 may be any of other machines or various peripheral devices (eg, a peripheral device coupled via USB).

[0052] In addition, the communication component 636 can detect an identifier or include components operable to detect an identifier. For example, the communication component 636 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via the communication component 636, such as location via Internet Protocol (IP) geolocation, location via Internet Protocol (IP), location information ... Location of signal triangulation, location of NFC beacon signals that can indicate a specific location via detection, etc.

[0053] Various memories (e.g., main memory 612, static memory 614, and memory of processor 602) and storage unit 616 may store one or more sets of instructions and data structures (e.g., software) embodying or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 608), when executed by processor 602, cause various operations to implement the disclosed examples.

[0054] The instructions 608 may be sent or received over the network 620 via a network interface device (e.g., a network interface component included in the communications component 636) using a transmission medium and using any of several well-known transmission protocols, such as the Hypertext Transfer Protocol (HTTP). Similarly, the instructions 608 may be sent or received via a coupling (e.g., a peer-to-peer coupling) to the device 622 using a transmission medium.

[0055] Glossary

[0056] "Carrier signal" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine and includes digital or analog communication signals or other intangible media to facilitate communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0057] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistant (PDA), smartphone, tablet computer, ultrabook, netbook, laptop computer, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or any other communications device that a user may use to access a network.

[0058] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Plain Old Telephone Service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long distance protocols, or other data transmission technologies.

[0059] "Component" refers to a device, physical entity or logic with the following boundaries, which are defined by function or subroutine calls, branch points, APIs or other technologies that provide partitioning or modularization for specific processing or control functions. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components, and can be part of a program that generally performs a specific function among related functions. A component can constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured as a hardware component for performing certain operations described herein by software (e.g., an application or an application part). Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component can be a dedicated processor such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) that is uniquely customized to perform the configured function, rather than a general-purpose processor. It will be understood that it can be decided to implement the hardware component mechanically, in a dedicated and permanently configured circuit or in a temporarily configured circuit (for example, configured by software) for cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (for example, hardwired) or temporarily configured (for example, programmed) to operate in some way or perform certain operations described herein. Considering that the hardware component is temporarily configured (for example, programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any one time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to, for example, constitute a specific hardware component at one time and constitute different hardware components at different times. A hardware component can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled.When there are multiple hardware components at the same time, communication can be achieved by signal transmission between or among two or more hardware components (for example, by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process the stored output. The hardware component can also initiate communication with an input device or an output device and can operate on resources (for example, the collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (for example, by software) or permanently configured to perform the relevant operations. Whether temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein specific one or more processors are examples of hardware. For example, at least some of the operations of the method can be performed by the parts implemented by one or more processors 1004 or processors. In addition, the one or more processors can also be operated to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some of the operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations can be distributed between processors, is not only resident in a single machine, but is deployed across multiple machines. In some example embodiments, a processor or the parts implemented by a processor can be located in a single geographic location (for example, in a home environment, an office environment or a server group). In other example embodiments, a processor or the parts implemented by a processor can be distributed across several geographic locations.

[0060] "Computer-readable storage media" refers to both machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0061] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0062] “Non-transitory computer-readable storage medium” refers to a tangible medium capable of storing, encoding, or carrying instructions for execution by a machine.

[0063] "Signal medium" refers to any intangible medium that can store, encode, or carry instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be taken to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and are used interchangeably in this disclosure.

[0064] Additionally, the present technology can be configured as follows.

[0065] (1) A method comprising: maintaining, at a backend server, an augmented reality component identified by an augmented reality component identifier, the backend server providing a message client implemented by one or more processors of a client device, the augmented reality component being configured to modify content captured by a camera provided at the client device according to values ​​of one or more launch attributes; providing, at the backend server, a deep link, the deep link comprising a payload for storing the augmented reality component identifier and the values ​​of the one or more launch attributes; and at the backend server, in response to detecting activation of the deep link from a third-party resource executed at the client device, performing the following operations: extracting corresponding values ​​of the one or more launch attributes from the payload of the deep link and storing the extracted corresponding values ​​so that the message client launches to a camera view screen indicating that the camera is activated, using the corresponding values, dynamically configuring the augmented reality component to modify content captured by the camera by applying modifications according to the corresponding values ​​of the one or more launch attributes, and loading the dynamically configured augmented reality component into the message client.

[0066] (2) The method of (1), wherein storing the extracted corresponding value comprises persisting the extracted corresponding value at a persistent storage system until another instance of the deep link is activated from the third-party resource executed on the client device.

[0067] (3) The method of (1), wherein: the value of an attribute from the one or more startup attributes is an alphanumeric string; and the modification is superimposing the alphanumeric string on a specified area of ​​the content captured by the camera device.

[0068] (4) The method of (1), wherein the value of a property from the one or more startup properties is a reference to a binary asset.

[0069] (5) The method of (4), wherein the binary asset indicates a persistent storage location, the method comprising: storing the binary asset at the persistent storage location at the client device.

[0070] (6) The method of (4), wherein the binary asset is a three-dimensional machine learning model.

[0071] (7) The method of (6), wherein applying the modification comprises executing the three-dimensional machine learning model using the content captured by the camera device as input to the three-dimensional machine learning model.

[0072] (8) The method according to (1) further includes: activating the augmented reality component represented by the augmented reality component identifier, wherein the activation causes the modification identified by the corresponding value of the one or more startup attributes to be applied to the content captured by the camera device.

[0073] (9) The method according to (1) further includes: in response to detecting activation of another instance of the deep link from the third-party resource executed at the client device, the payload of the other instance of the deep link includes a new value: overwriting the stored corresponding value with the new value, and dynamically configuring the augmented reality component using the new value.

[0074] (10). The method according to (1) further includes: detecting a request from the message client to transmit the content captured by the camera device and modified by the dynamically configured augmented reality component to another computing device; and transmitting the content captured by the camera device and modified by the dynamically configured augmented reality component to the other computing device.

[0075] (11). A system comprising: one or more processors; and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium comprising instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations comprising: maintaining, at a back-end server, an augmented reality component identified by an augmented reality component identifier, the back-end server providing a message client implemented by one or more processors of a client device, the augmented reality component being configured to modify content captured by a camera provided at the client device according to the value of one or more startup attributes, the client device hosting the message client; providing, at the back-end server, a deep link, the deep link comprising a method for storing the said augmented reality component identifier and a payload of the values ​​of the one or more launch attributes; and at the back-end server, in response to detecting activation of the deep link from a third-party resource executed at the client device, performing the following operations: extracting the corresponding values ​​of the one or more launch attributes from the payload of the deep link, and storing the extracted corresponding values, so that the message client is launched to a camera view screen, the camera view screen indicates that the camera is activated, using the corresponding values, dynamically configuring the augmented reality component to modify the content captured by the camera by applying modifications according to the corresponding values ​​of the one or more launch attributes, and loading the dynamically configured augmented reality component in the message client.

[0076] (12) A system according to (11), wherein storing the extracted corresponding value includes: persisting the extracted corresponding value at a persistent storage system until another instance of the deep link is activated from the third-party resource executed on the client device.

[0077] (13) A system according to (11), wherein: the value of the attribute from the one or more startup attributes is an alphanumeric string; and the modification is superimposing the alphanumeric string on a specified area of ​​the content captured by the camera device.

[0078] (14) The system of (11), wherein the value of a property from the one or more startup properties is a reference to a binary asset.

[0079] (15) The system of (14), wherein the binary asset indicates a persistent storage location, the method comprising: at the client device, storing the binary asset at the persistent storage location.

[0080] (16) The system of (14), wherein the binary asset is a three-dimensional machine learning model.

[0081] (17). A system according to (16), wherein applying the modification includes: executing the three-dimensional machine learning model using the content captured by the camera device as input to the three-dimensional machine learning model.

[0082] (18). According to the system described in (11), the operation also includes: activating the augmented reality component represented by the augmented reality component identifier, and the activation causes the modification identified by the corresponding value of the one or more startup attributes to be applied to the content captured by the camera device.

[0083] (19). According to the system described in (11), the operation also includes: in response to detecting the activation of another instance of the deep link from the third-party resource executed at the client device, the payload of the other instance of the deep link includes a new value: overwriting the stored corresponding value with the new value, and dynamically configuring the augmented reality component using the new value.

[0084] (20) A machine-readable non-transitory storage medium having instruction data, the instruction data being executable by a machine so that the machine performs operations including: maintaining an augmented reality component identified by an augmented reality component identifier at a backend server, the backend server providing a message client implemented by one or more processors of a client device, the augmented reality component being configured to modify content captured by a camera provided at the client device according to the value of one or more startup attributes; providing a deep link at the backend server, the deep link including a method for storing the augmented reality component identifier and the one or more startup attributes ; and at the back-end server, in response to detecting activation of the deep link from a third-party resource executed at the client device, performing the following operations: extracting the corresponding values ​​of the one or more launch attributes from the payload of the deep link, and storing the extracted corresponding values, so that the message client is launched to a camera view screen, the camera view screen indicates that the camera is activated, using the corresponding values, dynamically configuring the augmented reality component to modify the content captured by the camera by applying modifications according to the corresponding values ​​of the one or more launch attributes, and loading the dynamically configured augmented reality component in the message client.

Claims

1. A system for dynamic augmented reality, the system comprising: one or more processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting, at a backend server providing a messaging client implemented by one or more processors of the device, activation of a deep link from a third-party resource executing at a device, the backend server providing a messaging client implemented by one or more processors of the device, the third-party resource provided by an entity different from a provider of the backend server; Obtain corresponding values ​​of one or more launch properties for the augmented reality component from the deep link's payload; as well as The message client is started at the device and the augmented reality component is loaded into the message client.

2. The system according to claim 1, wherein: The augmented reality component will modify content captured by the device's camera based on the values ​​of the one or more activation properties.

3. The system according to claim 2, wherein: The value from the corresponding value is text or an image.

4. The system according to claim 3, wherein: The augmented reality component is configured to superimpose the value on a portion of an image captured by the camera.

5. The system according to claim 1, wherein: The augmented reality component is maintained by the backend server.

6. The system according to claim 1, wherein: The augmented reality component is identified at the backend server by a first identifier.

7. The system according to claim 6, wherein: The payload of the deep link includes the first identifier.

8. The system of claim 7, comprising storing the first identifier and the corresponding value, wherein The storing includes maintaining the corresponding value at a persistent storage system until another instance of the deep link is activated from the third-party resource executing at the device.

9. The system according to any one of claims 1 to 8, wherein: Starting the message client is to enter a camera view screen, and the camera view screen indicates that the camera is activated.

10. The system according to any one of claims 1 to 8, wherein: The corresponding value in the payload of the deep link is generated by the third-party resource.

11. A system, comprising: one or more processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting, at a server, a request from a third-party resource executing at a user system, the third-party resource being provided by an entity different from a provider of the server; obtaining corresponding values ​​for one or more properties of the augmented reality component from the request; causing, at the user system, the augmented reality component to be loaded into an interactive application executing at the user system; as well as The interactive application is caused, at the user system, to display content captured by a camera of the user system modified by the augmented reality component according to the values ​​of the one or more properties.

12. A system comprising: one or more processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: detecting, at a backend server, a request from a third-party resource executing at a user system, the backend server providing an interactive application implemented by one or more processors of the user system, the third-party resource being provided by an entity different from a provider of the backend server; obtaining, from the request, corresponding values ​​of one or more activation properties for the augmented reality component; as well as The interactive application is started at the user system and the augmented reality component is loaded into the interactive application.

Images