Method, system and machine-readable non-transitory storage medium for dynamic augmented reality

Abstract

The technical problem of adding content originating from a third-party app to an augmented reality component is solved by configuring the augmented reality component maintained by a message server system to modify content captured by a camera device according to the value of one or more attributes, and allowing the values ​​to be delivered from the third-party app to the message server system in the payload of a deep link that references the ID of the augmented reality component. The content originating from the third-party app and provided to the augmented reality component maintained by the message server system can be dynamic, because content such as sports scores provided by a sports news channel app can change over time.

Description

[0001] Priority declaration

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 037,559, filed on June 10, 2020, and U.S. Patent Application Serial No. 16 / 948,268, filed on September 10, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to facilitating interaction between messaging clients and third-party resources. Background Art

[0004] The popularity of computer-implemented programs that allow users to access content online and interact with other users continues to grow. There are various computer-implemented applications that allow users to share content with other users through a messaging client. Some of such computer-implemented applications (referred to as apps) can be designed to run on mobile devices such as phones, tablets, or watches, while having back-end services provided on a server computer system to perform operations that may require more resources than reasonable resources to perform at the client device (e.g., storing large amounts of data or performing computationally expensive processing). For example, a messaging client and an associated messaging server system can be configured to allow online users to share content. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the highest digit or digits in the reference numeral refer to the figure number in which the element is first introduced. Some embodiments are shown by way of example and not limitation in the accompanying drawings, in which:

[0006] Figure 1 is a diagrammatic representation of a networked environment in which a developer tool system may be deployed, according to some examples.

[0007] Figure 2 is a block diagram of the architecture of a system for providing dynamic augmented reality components according to some examples.

[0008] Figure 3 is a flowchart of a method for providing a dynamic augmented reality component according to some examples.

[0009] Figure 4 is a diagrammatic representation of example content captured from a camera view screen of a messaging client.

[0010] Figure 5is a diagrammatic representation of example content captured from a camera view of a messaging client modified using a dynamic augmented reality component according to some examples.

[0011] Figure 6 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. DETAILED DESCRIPTION

[0012] The messaging server system (which hosts the backend services of the associated messaging client) is configured to allow a user to capture images and videos using a camera provided to a client device (which hosts the messaging client) and share the captured content with other users via network communications. The messaging server system is also configured to provide an augmented reality (AR) component accessible via the messaging client, which can be used to modify the content captured by the camera, for example by overlaying pictures or animations on top of captured images or video frames and by adding three-dimensional (3D) effects, objects, characters, and transformations.

[0013] The AR component can be implemented using a programming language suitable for app development (such as JavaScript or Java). The AR component can be executed in the messaging client when the camera device records the video, so that each captured frame is modified in a manner determined by the specific AR component configuration. For example, the AR component can be configured to detect the head position of the person being captured by the video in each frame of the video, and superimpose the image of the party hat on the detected head position in each frame, so that the viewer of the video captured using the AR component will see the person being captured in the video wearing a party hat. When the camera view screen is opened in the messaging client, the effects of the AR component can be applied to the captured image, and the messaging client can present one or more visual controls to the user, which allow the application and removal of visual effects generated by the AR component, as well as the selection of the desired AR component from multiple AR components.

[0014] The user can start the AR component by opening the message client (in some embodiments, opening to the camera view screen that is ready to capture or has captured an image or video) and selecting the icon representing the desired AR component. The AR component is identified by the corresponding AR component identifier in the message server system. In some embodiments, a specific AR component maintained by the message server system can be directly accessed from a third-party resource such as a third-party app via a deep link that references the AR component identifier, without having to first launch the message client (whether a stand-alone app or a web-based version of the app). In the context of the app, a deep link can be described as using a uniform resource identifier (URI) linked to a specific location within the app, rather than simply launching the app. When such a deep link is activated from a third-party app executed at a client device equipped with a camera, the deep link is passed to the message server system and returned to the real-time camera of the client device, where the AR component referenced by the AR component identifier in the deep link is activated. For the purpose of this description, activating the AR component in the message client causes the AR component to be available for user selection. Launching the AR component will cause the effect of the AR component to be applied to the captured content. Activating the live camera together with the operation of launching a given AR component causes the effect of the given AR component to be applied to the content being captured by the camera. In addition to providing access to the AR component directly from the third-party app, the message server system can be configured to allow third-party developers to share content originating from their third-party app to the AR component maintained by the message server system.

[0015] The technical problem of adding content originating from a third-party app to an AR component maintained by a message server system is solved by configuring the AR component to modify content captured by a camera device according to the value of one or more launch properties, and allowing the value to be delivered from the third-party app to the message server system in the payload of a deep link that references an AR component identifier of the AR component. The content provided to the AR component maintained by the message server system that originates from its third-party app (e.g., sports scores provided by a sports news channel app or current weather information provided by a weather app) may be dynamic in that the content may change over time.

[0016] For the purpose of this description, an AR component configured to modify the content captured by the camera according to the value of one or more startup attributes is referred to as a dynamic augmented reality component or a dynamic AR component. The modification type specified by the startup attribute may indicate that a certain type of content (e.g., text, image, or animation) will be superimposed on a portion of the captured image (e.g., centered on the lower right quadrant of the frame area). The value of such a startup attribute may be, for example, text, an image, or an animation. For example, the value of the startup attribute provided from a sports news channel app may be an image or string indicating the cumulative score of an ongoing sports event. For the purpose of this description, the value of the startup attribute that can be delivered to the message server system via a deep link may be simply referred to as an attribute. Another example of a startup attribute is a startup attribute that references a binary asset (e.g., a three-dimensional machine learning model that uses content captured by a camera as input), which can be executed relative to the captured content, resulting in the entire object in the captured frame (e.g., a human body depicted in the frame) being replaced with a three-dimensional (3D) mesh or texture.

[0017] The message server system is configured to provide a set of developer tools to the user, which includes features that allow third-party app developers to include deep links in their third-party apps, which reference the AR component identifier and can include the value of one or more launch attributes in the payload. The developer tools provided by the message server system can be called from a third-party computer system via a developer portal, which can be accessed via a web browser. The developer portal can be downloaded to a third-party computer system, in which case a web browser may not be required. The developer portal can be used to obtain a deep link that references a given AR component identifier, and to access an AR component service application programming interface (API), which is configured to allow third-party developers to create new dynamic AR components. In some embodiments, the message server system pushes the AR component template and the properties and assets that determine the functionality of the new AR component to the developer portal via the AR component service API. A user interface (UI) provided with a developer portal allows third-party developers to select any attribute for the new AR component and specify at least one attribute whose value will be received from the third-party app via a deep link.

[0018] In operation, a third-party app uses developer tools provided by the messaging server system to construct a request to link to an AR component with a specific AR component identifier. In some examples, the third-party app is a registered client with respect to the developer tools provided by the messaging server system. The developer tools system is constructing a deep link and encrypting a payload that stores the AR component identifier and properties ready for launch.

[0019] When the message server system detects activation of a deep link from a client device (e.g., when a user activates the deep link using associated visual controls provided by a third-party app's UI), the message server system decrypts the deep link payload and obtains the AR component identifier, verifies that the AR component referenced by the AR component identifier in the deep link is valid, and verifies that the AR component is unlockable (which means that it can be activated and used immediately in the message app). Properties are then extracted from the payload of the deep link. The extracted properties are stored in a specified data object associated with the AR component identifier at a persistent storage system. The data object is configured, for example, by a developer using a scripting language via a developer portal 132 to be populated with properties from the payload of the deep link when the associated AR component is initialized. When the AR component is started, the stored properties are exposed to the AR component using the data object. The persistent storage system can be associated with the message server system, or it can reside at a client computer system hosting the message client.

[0020] Once the AR component is activated, the content captured from the camera view is enhanced with modifications provided by the AR component. In some embodiments, sending or publishing the captured content enhanced with modifications provided by the AR component enables the messaging client to have the AR component available to the user for a predetermined period of time (e.g., 48 hours). The attributes extracted from the payload of the deep link and stored in the persistent storage system remain in the persistent storage system until the next time the deep link is enabled from the third-party app, at which time these stored attributes are overwritten by new attributes, which are the attributes extracted from the link at that time. In the case where the attributes extracted from the deep link are references to binary assets, such attributes indicate the storage location of the referenced binary asset, which may be at the client computer device hosting the messaging client.

[0021] The method described herein can be beneficially used for various third-party resources. Third-party resources can be understood as installed apps, as well as HTML-based mini-apps and games. HTML is a markup language for structuring and presenting content on the World Wide Web. An installed app is a software file that is installed on a device using executable code and can be launched independently of a message client. Mini-apps and games (which can be considered as tiny apps that can be downloaded to a client device and can run within a large app) are HTML-based apps that require a message client to download a markup language document and present the markup language document in a browser (in a web view) run by the message client. When launched by a user, an HTML file is retrieved from a server for presentation of the app.

[0022] Figure 11 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each of which hosts several applications including a messaging client 104 and a third-party app 103. The third-party app 103 is configured to allow a user to access functionality provided by a third-party system 130. Each messaging client 104 is communicatively coupled to a messaging server system 108 and other instances of the messaging client 104 via a network 106 (e.g., the Internet).

[0023] The message client 104 is able to communicate and exchange data with another message client 104 and a message server system 108 via the network 106. The data exchanged between the message clients 104 and between the message clients 104 and the message server system 108 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video or other multimedia data).

[0024] The message server system 108 is capable of providing server-side functionality to a particular message client 104 via the network 106. Although certain functions of the message system 100 are described herein as being performed by the message client 104 or by the message server system 108, the location of certain functions within the message client 104 or within the message server system 108 may be a design choice. For example, it may be technically preferred that certain technologies and functions may be initially deployed within the message server system 108, but later migrated to a message client application 104 where the client device 102 has sufficient processing power.

[0025] The message server system 108 supports various services and operations provided to the message clients 104. Such operations include sending data to the message clients 104, receiving data from the message clients 104, and processing data generated by the message clients 104. As examples, the data may include message content, client device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchanges within the message system 100 are activated and controlled by functions available through the user interface (UI) of the message client 104.

[0026] Turning now specifically to the message server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118 that facilitates access to a database 120. A web server 124 is coupled to the application server 112 and provides a web-based interface to the application server 112. To this end, the web server 124 processes incoming network requests via the hypertext transfer protocol (HTTP) and several other related protocols. The database 120 stores data associated with messages processed by the application server 112, such as profile data about a particular entity. In the case where the entity is a person, the profile data includes, for example, the user's name, notification and privacy settings, and the user's self-reported age and records related to changes made by the user to his or her profile data.

[0027] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the message client 104 can call or query to activate the functions of the application server 112. The API server 110 exposes various functions supported by the application server 112, including: account registration; login functions; sending messages from a particular message client 104 to another message client 104 via the application server 112; sending media files (e.g., images or videos) from the message client 104 to the message server system 114 for possible access by another message client 104; opening application events (e.g., related to the message client 104); and various functions supported by developer tools provided by the message server system 108 for use by third-party computer systems.

[0028] The application server 112 hosts a number of server applications and subsystems, including, for example, a message server system 114, an image processing server 116, and a social network server 122. The message server system 114 implements a number of message processing technologies and functions, particularly related to aggregating and otherwise processing content (e.g., text and multimedia content) included in messages received from multiple instances of the message client 104. The image processing server 116 is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the message server system 114. The social network server 122 supports a variety of social networking functions and services and makes them available to the message server system 114.

[0029] Figure 11. The developer tool server 117 is also shown in the figure. The developer tool server 117 maintains one or more software developer kits (SDKs) that allow users to integrate some features provided by the message server system across their apps (also referred to as third-party apps). These features include deep links to dynamic AR components, which are configured to modify the content captured by the camera device according to the value of one or more launch attributes delivered from the third-party app to the message server system in the payload of the deep link of the AR component identifier that references the AR component. The functions provided by the developer tool server 117 can be accessed from a third-party computer system via a developer portal that can be accessed via a web browser. In some examples, a developer portal that provides access to the functions provided by the developer tool server 117 to a third-party computer system (e.g., a third-party system 130) can be downloaded to the third-party computer system, in which case a web browser may not be required. The third-party system 130 is shown to include a developer portal 132. As described above, the developer portal 132 can be accessed via a web browser executed on the third-party app provider system 130, or downloaded to the third-party computer system 130.

[0030] Figure 2 2 is a block diagram illustrating an example system 200 for providing a dynamic AR component. The system 200 includes a deep link generator 210, a deep link activation detector 220, an attribute extractor 230, and an augmented reality component activator 240. As mentioned above, for the purposes of this description, an augmented reality component may be referred to as an AR component. The deep link generator 210 is configured to construct a deep link that includes a method for storing a representation of the Figure 1 The message server system 108 includes an identifier of the AR component (augmented reality component identifier) ​​and a payload for storing the value of one or more activation attributes. As explained above, the AR component is configured to modify the content captured by the camera device according to the value of one or more activation attributes. The deep link activation detector 220 is configured to detect the activation of a third-party resource (e.g., from a client device) executed at the client device. Figure 1 The third-party app 103 executed on the client device 102 of the present invention) activates the deep link. The attribute extractor 230 is configured to extract corresponding values ​​of one or more launch attributes from the payload of the deep link and store the extracted corresponding values ​​in the persistent storage system.

[0031] The augmented reality component activator 240 is configured to cause the messaging client 104 to launch to a camera view screen. The camera view screen indicates that the camera is activated. The augmented reality component activator 240 dynamically configures the AR component using the corresponding value when the corresponding value becomes available or updated in the persistent storage system. After the AR component is dynamically configured, the AR component is ready to modify the captured content according to the stored corresponding value. The augmented reality component activator 240 then loads the dynamically configured AR component into the messaging client. Each of the various components of the system 200 can be set in Figure 1 The client device 102 and / or the message server system 108. Additional details regarding the operation of the system 200 are provided below.

[0032] Figure 3 is a flow chart of a method 300 for providing a dynamic AR component. The method 300 may be performed by processing logic, which may include hardware (e.g., dedicated logic, programmable logic, microcode, etc.), software, or a combination thereof. In one example implementation, some or all of the processing logic resides in Figure 1 The client device 102 and / or Figure 1 At the message server system 108 of the client device. At operation 310, the message server system 108 detects a request from a third-party resource (e.g., from a Figure 1 The method of claim 31 is to activate a deep link by a third-party app 103 executed at a client device 102 of the messaging server system 108. The deep link is constructed to include in its payload an AR component identifier representing an AR component in the messaging server system 108 and the value of one or more launch attributes. At operation 320, the values ​​of one or more launch attributes are extracted and stored in a persistent storage system. As mentioned above, the persistent storage system may reside at the client device 103 or the messaging server system 108. At operation 330, the messaging client 104 is caused to launch to a camera view screen. At operation 340, the corresponding value is retrieved from the persistent storage system, and the AR component is configured using the corresponding value so that it can be used to modify the content captured by the camera according to the stored corresponding value. At operation 350, the configured AR component is loaded into the messaging client.

[0033] Although the described flow chart may show the operation as a sequential process, many operations in the operation can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. When its operation is completed, the process terminates. The process can correspond to a method, a program, an algorithm, etc. The operation of the method can be performed in whole or in part, can be performed in combination with some or all operations in other methods, and can be performed by any number of different systems (such as the system described herein) or any part thereof (such as a processor included in any system).

[0034] Figure 4 4 is a diagrammatic representation of example content captured from a camera view of a messaging client 104. Shown in area 410 is a person captured in a still image or a frame of a video. Visual controls 420 can be activated to apply modifications of the associated AR component to the captured content. In this example, visual controls 420 are executable to enhance the captured content with a current sports score obtained from a third-party app 103 from which the messaging client 104 was launched via a deep link. Figure 5 is a diagrammatic representation 500 of content captured from a camera view of a messaging client, modified using an AR component referenced in a deep link and based on a value extracted from the payload of the deep link (in this example, the current sports score). Shown in area 510 are: a person captured in a still image or frame of a video; and also shown in area 530 as superimposed on the captured content. Visual controls 520 are shown using a thicker black line to indicate that the associated AR component is activated. The deep link is in Figure 1 The process of constructing a deep link includes additional variables of the payload of the deep link. The following is an example 1 showing the generated variables included in the payload of the deep link. Figure 5 Example attributes of the current sports game score are shown in area 530 of the diagram - variables team_one_score, team_two_score, team_one_name, team_two_name and their associated values.

[0035] Example 1

[0036] team_one_score:5, team_two_score:3, team_one_name:

[0037] "Team Beagle", team_two_name: "Team Beaver"

[0038] The properties shown in Example 1 are stored in a data object associated with the AR component identifier referenced in the deep link. When the AR component is loaded into the messaging client 104, these properties are passed directly from the data object to the AR component. In one embodiment, using Figure 5 , the AR component can be configured to modify binary assets within the AR component. For example, the AR component can be configured to highlight the team name and score using team colors, and can also be configured to show the associated team logo. The team colors and logo images can be predefined in the AR component and selected based on properties provided in the deep link payload.

[0039] Machine Architecture

[0040] Figure 6 600 in which instructions 608 (e.g., software, programs, applications, applet, app, or other executable code) may be executed to cause the machine 700 to perform any one or more of the methods discussed herein. For example, the instructions 608 may cause the machine 600 to perform any one or more of the methods described herein. The instructions 608 convert the general non-programmed machine 600 into a specific machine 600 that is programmed to perform the functions described and shown in the manner described. The machine 600 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 600 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 608 specifying actions to be taken by the machine 600. In addition, although only a single machine 600 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 1100 to perform any one or more of the methods discussed herein. For example, the machine 600 may include any of the client device 102 or a number of server devices that form part of the message server system 108. In some examples, the machine 600 may also include both a client and server system, where certain operations of a particular method or algorithm are performed on the server side and certain operations of a particular method or algorithm are performed on the client side.

[0041] The machine 600 may include a processor 602, a memory 604, and an input / output (I / O) component 638 that may be configured to communicate with each other via a bus 640. In an example, the processor 602 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 606 that executes instructions 608 and a processor 610. The term "processor" is intended to include a multi-core processor that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 6 Multiple processors 602 are shown, but machine 600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0042] The memory 604 includes a main memory 612, a static memory 614, and a storage unit 616 that are accessible to the processor 602 via the bus 640. The main memory 604, the static memory 614, and the storage unit 616 store instructions 608 that implement any one or more of the methods or functions described herein. The instructions 608 may also reside, completely or partially, within the main memory 612, within the static memory 614, within a machine-readable medium 618 within the storage unit 616, within at least one of the processors 602 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 600.

[0043] The I / O components 638 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 638 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It will be appreciated that the I / O components 638 may include Figure 6624 and 626. In various examples, the I / O components 638 may include a user output component 624 and a user input component 626. The user output component 624 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 626 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen that provides the location and force of a touch or touch gesture, or other tactile input component), an audio input component (e.g., a microphone), etc.

[0044] In other examples, the I / O component 638 may include a biometric component 628, a motion component 630, an environmental component 632, or a positioning component 634, as well as various other components. For example, the biometric component 628 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 630 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0045] Environmental components 632 include, for example, one or more cameras (with still image / photo and video capabilities), an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0046] With respect to cameras, client device 102 can have a camera system including, for example, a front-facing camera on a front surface of client device 102 and a rear-facing camera on a rear surface of client device 102. The front-facing camera can, for example, be used to capture still images and videos of a user of client device 102 (e.g., a "selfie"), which can then be enhanced with the enhancement data (e.g., filters) described above. For example, the rear-facing camera can be used to capture still images and videos in a more traditional camera mode, which are similarly enhanced using the enhancement data. In addition to the front and rear-facing cameras, client device 102 can also include a 360° camera for capturing 360° photos and videos.

[0047] Additionally, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even triple, quad, or quintuple rear camera configurations on the front and back sides of the client device 102. For example, these multi-camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0048] The location component 634 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or a barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), and the like.

[0049] A variety of technologies may be used to implement communications. The I / O components 638 also include a communication component 636 that is operable to couple the machine 600 to the network 620 or device 622 via a corresponding coupling or connection. For example, the communication component 636 may include a network interface component or another suitable device to interface with the network 620. In other examples, the communication component 636 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low power consumption), Device 622 may be any peripheral device (eg, a peripheral device coupled via USB) from another machine or from a variety of peripheral devices.

[0050] In addition, the communication component 636 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 636 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D bar codes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via the communication component 636, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0051] Various memories (e.g., main memory 612, static memory 614, and memory of processor 602) and storage unit 616 may store one or more sets of instructions and data structures (e.g., software) embodying or used by any one or more of the various methods or functions described herein. These instructions (e.g., instructions 608) when executed by processor 602 cause various operations to implement the disclosed examples.

[0052] The instructions 608 may be sent or received over the network 620 via a network interface device (e.g., a network interface component included in the communications component 636) using a transmission medium and using any of several well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 608 may be sent or received via a coupling (e.g., a peer-to-peer coupling) to the device 622 using a transmission medium.

[0053] Glossary

[0054] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0055] "Client Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0056] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, other type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0057] "Component" refers to a device, physical entity or logic with the following boundaries, which is defined by function or subroutine calls, branch points, APIs or other technologies that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be packaged functional hardware units designed for use with other components, and can be part of a program that generally performs specific functions in related functions. Components can constitute software components (e.g., codes embodied on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured as hardware components for performing certain operations described herein by software (e.g., applications or application parts). Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component can be a dedicated processor such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that it can be decided to implement the hardware component mechanically, in a dedicated and permanently configured circuit or in a temporarily configured (e.g., configured by software) circuit for cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an embodiment in which the hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured as different special-purpose processors (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly to constitute a specific hardware component, for example, at one time, and to constitute different hardware components at different times. Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components (e.g., by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process the stored output. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors 1004 or the parts implemented by the processor. In addition, the one or more processors can also be operated to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some example embodiments, a processor or a part implemented by a processor can be located in a single geographic location (for example, in a home environment, an office environment or a server group). In other example embodiments, a processor or a part implemented by a processor can be distributed across several geographic locations.

[0058] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0059] “Machine storage media” refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage media,” “device storage media,” and “computer storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine storage media,” “computer storage media,” and “device storage media” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term “signal media.”

[0060] “Non-transitory computer-readable storage medium” refers to tangible media capable of storing, encoding, or carrying instructions to be executed by a machine.

[0061] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.

Claims

1. A method for dynamically augmenting reality, comprising: maintaining, at a backend server, an augmented reality component identified by an augmented reality component identifier, the backend server providing a message client implemented by one or more processors of a client device, the augmented reality component being configured to modify content captured by a camera provided at the client device according to values ​​of one or more activation attributes; At the backend server, providing a deep link, the deep link comprising a payload for storing the augmented reality component identifier and the values ​​of the one or more launch attributes; as well as At the backend server, in response to detecting activation of the deep link from a third-party resource executed at the client device, performing the following operations: extracting corresponding values ​​of the one or more launch attributes from the payload of the deep link and storing the extracted corresponding values, The message client is started to a camera view screen, wherein the camera view screen indicates that the camera is activated. using the corresponding values, dynamically configuring the augmented reality component to modify content captured by the camera by applying modifications in accordance with the corresponding values ​​of the one or more enabling properties, and The dynamically configured augmented reality component is loaded into the messaging client.

2. The method according to claim 1, wherein: Storing the extracted corresponding value includes persisting the extracted corresponding value at a persistent storage system until an additional instance of the deep link is activated from the third-party resource executing on the client device.

3. The method according to claim 1, wherein: a value of an attribute from the one or more startup attributes is an alphanumeric string; and The modification is superimposing the alphanumeric string on a designated area of ​​the content captured by the camera.

4. The method according to claim 1, wherein: The value of a property from the one or more launch properties is a reference to a binary asset.

5. The method according to claim 4, wherein: The binary asset indicates a persistent storage location, and the method includes, at the client device, storing the binary asset at the persistent storage location.

6. The method according to claim 4, wherein: The binary asset is a three-dimensional machine learning model.

7. The method according to claim 6, wherein: Applying the modification includes executing the three-dimensional machine learning model using the content captured by the camera device as input to the three-dimensional machine learning model.

8. The method according to claim 1, further comprising: The augmented reality component represented by the augmented reality component identifier is activated, the activation causing the modification identified by the corresponding values ​​of the one or more activation properties to be applied to the content captured by the camera.

9. The method according to claim 1, further comprising: In response to detecting activation of an additional instance of the deep link from the third-party resource executing at the client device, the payload of the additional instance of the deep link includes a new value: overwriting the corresponding stored value with the new value, and The augmented reality component is dynamically configured using the new value.

10. The method according to claim 1, further comprising: detecting a request from the messaging client to transmit content captured by the camera and modified by the dynamically configured augmented reality component to another computing device; as well as Content captured by the camera and modified by the dynamically configured augmented reality component is transmitted to the additional computing device.

11. A system for dynamic augmented reality, comprising: one or more processors; as well as A non-transitory computer-readable storage medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: maintaining, at a backend server, an augmented reality component identified by an augmented reality component identifier, the backend server providing a message client implemented by one or more processors of a client device, the augmented reality component being configured to modify content captured by a camera provided at the client device according to values ​​of one or more launch attributes, the client device hosting the message client; At the backend server, providing a deep link, the deep link comprising a payload for storing the augmented reality component identifier and the values ​​of the one or more launch attributes; as well as At the backend server, in response to detecting activation of the deep link from a third-party resource executed at the client device, performing the following operations: extracting corresponding values ​​of the one or more launch attributes from the payload of the deep link and storing the extracted corresponding values, The message client is started to a camera view screen, wherein the camera view screen indicates that the camera is activated. using the corresponding values, dynamically configuring the augmented reality component to modify content captured by the camera by applying modifications in accordance with the corresponding values ​​of the one or more enabling properties, and The dynamically configured augmented reality component is loaded into the messaging client.

12. The system according to claim 11, wherein: Storing the extracted corresponding value includes persisting the extracted corresponding value at a persistent storage system until an additional instance of the deep link is activated from the third-party resource executing on the client device.

13. The system of claim 11, wherein: a value of an attribute from the one or more startup attributes is an alphanumeric string; and The modification is superimposing the alphanumeric string on a designated area of ​​the content captured by the camera.

14. The system according to claim 11, wherein: The value of a property from the one or more launch properties is a reference to a binary asset.

15. The system of claim 14, wherein: The binary asset indicates a persistent storage location, and the method includes, at the client device, storing the binary asset at the persistent storage location.

16. The system of claim 14, wherein: The binary asset is a three-dimensional machine learning model.

17. The system of claim 16, wherein: Applying the modification includes executing the three-dimensional machine learning model using the content captured by the camera device as input to the three-dimensional machine learning model.

18. The system of claim 11, the operations further comprising: The augmented reality component represented by the augmented reality component identifier is activated, the activation causing the modification identified by the corresponding values ​​of the one or more activation properties to be applied to the content captured by the camera.

19. The system of claim 11, the operations further comprising: In response to detecting activation of the additional instance of the deep link from the third-party resource executing at the client device, the payload of the additional instance of the deep link includes a new value: overwriting the corresponding stored value with the new value, and The augmented reality component is dynamically configured using the new value.

20. A machine-readable non-transitory storage medium having instruction data, the instruction data being executable by a machine to cause the machine to perform operations including: maintaining, at a backend server, an augmented reality component identified by an augmented reality component identifier, the backend server providing a message client implemented by one or more processors of a client device, the augmented reality component being configured to modify content captured by a camera provided at the client device according to values ​​of one or more activation attributes; At the backend server, providing a deep link, the deep link comprising a payload for storing the augmented reality component identifier and the values ​​of the one or more launch attributes; and At the backend server, in response to detecting activation of the deep link from a third-party resource executed at the client device, performing the following operations: extracting corresponding values ​​of the one or more launch attributes from the payload of the deep link and storing the extracted corresponding values, The message client is started to a camera view screen, wherein the camera view screen indicates that the camera is activated. using the corresponding values, dynamically configuring the augmented reality component to modify content captured by the camera by applying modifications in accordance with the corresponding values ​​of the one or more enabling properties, and The dynamically configured augmented reality component is loaded into the messaging client.

Images