Automatically generating descriptions of augmented reality effects

Abstract

The computer system accesses a first image and a second image. The second image is generated by applying augmented reality (AR) effects to the first image. The computer system provides the first image, the second image, and a cue to a visual semantic machine learning model to obtain an output describing at least one feature of the AR effect. The computer system generates a description of the AR effect based on the output of the visual semantic machine learning model. The computer system stores the description of the AR effect in association with an identifier for the AR effect.

Description

[0001] Priority requirements

[0002] This application claims the benefit of priority to U.S. Patent Application Serial No. 18 / 502,679, filed November 6, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0003] The topics disclosed herein generally relate to augmented reality (AR) technology. More specifically, but not exclusively, the topics disclosed herein relate to the generation of descriptions of AR effects for use in interactive systems. Background Technology

[0004] Some interactive applications enable users to apply AR effects (also known as augmentations) to content items (such as images or videos captured by the user). AR effects can be stored in the interactive system along with metadata such as keywords. For example, the interactive system can link AR effects to relevant keywords, making it easier for users of the interactive application to search for and apply the desired AR effects. Attached Figure Description

[0005] In accompanying drawings that are not necessarily drawn to scale, the same reference numerals can describe similar parts in different views. To facilitate identification of any discussion of a particular element or action, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced. Some non-limiting examples are shown in the figures in the accompanying drawings:

[0006] Figure 1 It is a diagrammatic representation of a networked environment in which the content of this disclosure can be deployed, based on some examples.

[0007] Figure 2 It is a graphical representation of an interactive system with both client-side and server-side functionality, based on some examples.

[0008] Figure 3 It is a graphical representation of the data structures maintained in the database based on some examples.

[0009] Figure 4 It is a graphical representation of the components of the system based on some example AR effects.

[0010] Figure 5 This is a flowchart illustrating a method suitable for generating AR effects based on some examples.

[0011] Figure 6 The first image is shown based on some examples.

[0012] Figure 7 This demonstrates, based on some examples, how to apply AR effects. Figure 6 The second image is generated from the first image.

[0013] Figure 8 This illustrates, based on some examples, how to arrange spaces according to a predetermined plan. Figure 6 The first image and Figure 7 The stitched image is generated by stitching together the second image.

[0014] Figure 9 The first image is shown based on some examples.

[0015] Figure 10 This demonstrates, based on some examples, how to apply AR effects. Figure 9 The second image is generated from the first image.

[0016] Figure 11 The machine learning pipeline is illustrated using diagrams based on some examples.

[0017] Figure 12 The training and use of machine learning programs are illustrated using diagrams based on some examples.

[0018] Figure 13 It is a graphical representation based on some example messages.

[0019] Figure 14 The diagram illustrates a network environment in which a head-mounted wearable device can be implemented, based on some examples.

[0020] Figure 15 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0021] Figure 16 It is a block diagram showing an example of a software architecture that can be implemented within it. Detailed Implementation

[0022] As used in this disclosure, the term "AR effect" refers to an effect, modification, removal, addition, or combination thereof that alters an image or image sequence (e.g., a video clip including a sequence of image frames) when captured and rendered without any such effect, modification, removal, or addition. AR effects may also be referred to as "enhancements." Examples of AR effects may include two-dimensional or three-dimensional effects, filters, lenses, media overlays (e.g., text, color, or image overlays), AR experiences, extended reality (XR) experiences, or combinations thereof.

[0023] AR effects can provide special, interesting, entertaining, or useful enhancements to content items such as images or videos. For example, a user of an interactive application can select the AR effect to be applied to video content captured using the user's device's camera. The AR effect can then be applied in real time (e.g., before or during content capture, to objects presented in the camera feed interface) or after content capture (e.g., to a video file retrieved from a storage location associated with the user's device).

[0024] For example, AR effects can overlay images onto a person's face, place virtual objects in specific locations relative to real-world objects, or alter the colors of an image. In some cases, AR effects can include changes to the format or style of content items (e.g., applying a "green screen" effect to a video captured by a user). In others, AR effects can produce both aesthetic and formatting changes.

[0025] As mentioned, AR effects can be stored in interactive systems along with metadata such as keywords or "tags." In some cases, the metadata of a specific AR effect can provide a useful description of the AR effect, such as facilitating the search for AR effects, generating AR effect recommendations, and ranking AR effects. However, interactive systems can support a large number of AR effects (e.g., thousands or even millions of different AR effects), making the manual review and tagging of AR effects a challenging task. Therefore, it is expected that the generation of AR effect descriptions will be automated, rather than requesting human "tagners" to perform this task, for example.

[0026] It might be possible to automatically generate keywords or labels by using object recognition processing to augment images. For example, a machine learning model could be trained to detect objects in an image and output predicted labels or categories based on the detected objects. However, this technique may not produce the desired output. For instance, an object recognition machine learning model might correctly detect certain objects in an image that have been augmented with AR effects, but it might fail to distinguish between objects specifically associated with the AR effect and objects that form part of the original or base image. Therefore, automated descriptions of AR effects produced in this way may have limited usefulness or relevance.

[0027] The examples described in this paper can address or mitigate the technical problems mentioned above by using techniques involving visual semantic machine learning models. In some examples, visual semantic machine learning models are used to provide more accurate, relevant, or comprehensive descriptions of AR effects supported by interactive systems. The techniques described in this paper can allow the automatic generation of AR effect descriptions that separate the presented AR effect from the "base" or "background" content, making the descriptions more useful in downstream tasks such as AR effect indexing, searching, ranking, or categorizing. In some examples, AR effect descriptions can serve as annotations for AR effects within interactive systems or as labels for AR effects.

[0028] One method may include accessing a first image and a second image by a computer system. The second image is generated by applying an AR effect to the first image. For example, the first image may be part of a first video or a base video, and the second image may come from an enhanced video generated by applying an AR effect to the base video. The method may include obtaining the second image by applying an AR effect to the first image (e.g., to a video containing the first image) by a computer system.

[0029] Therefore, the first image can be a frame from the base video, and the second image can be a frame from the enhanced video. The method may include identifying frames from the enhanced video as corresponding to frames from the base video in time (e.g., at or near the same timestamp), extracting frames from the base video to obtain the first image, and extracting frames from the enhanced video to obtain the second image.

[0030] In some examples, the frames of the base video and the frames of the enhanced video do not correspond exactly in time. For example, the two videos may have unequal frame durations. Therefore, the term "corresponding in time" should be interpreted to include relatively small or slight differences. In some examples where the frames of the two videos have unequal durations, the frame duration of one video (with a longer duration) may be truncated to correspond to the frame duration of the other video (with a shorter duration).

[0031] In some examples, a first image, a second image, and one or more cues are fed to a visual semantic machine learning model to obtain an output describing at least one feature (e.g., a visual feature or visual effect) that describes the AR effect. A description of the AR effect is generated based on the output of the visual semantic machine learning model. In some examples, as further described below, the description is automatically generated by combining or aggregating the received responses to different cues.

[0032] The first and second images can be provided as a stitched image to a visual semantic machine learning model. In some examples, two image frames are stitched together to provide or simulate "before and after," which allows the visual semantic machine learning model to better analyze the effect of the augmentation. In some examples, more than two image frames can be used (e.g., by presenting a series of frames to illustrate the result of the AR effect).

[0033] As used herein, the term "stitched image" refers to any image formed by combining or integrating multiple (e.g., two or more) images or portions of multiple images. In some examples, the method involves generating a stitched image by stitching together a first image and a second image. A stitched image can be formed by positioning the first image and the second image relative to each other in a predetermined spatial arrangement to obtain the stitched image. Examples of predetermined spatial arrangements include horizontal stitching and vertical stitching.

[0034] One or more predefined cues can be selected to enable natural language-based feature extraction via a visual semantic machine learning model. The examples described in this paper allow for the generation of AR effect descriptions that include granular and / or full-scale details of the visual effects of the AR effects on features in content items. Predefined sets of cues can be applied to analyze multiple different AR effects to facilitate the automation of AR effect descriptions.

[0035] In some examples, multiple prompts are used to obtain descriptions of different features, dimensions, or aspects of the AR effect, or descriptions associated with different features, dimensions, or aspects of the AR effect. In some examples, a first image and a second image are provided to a visual semantic machine learning model along with at least a first prompt and a second prompt. The first and second prompts can request descriptions associated with different visual features of the AR effect. Descriptions of the AR effect can be generated based on the first and second outputs.

[0036] Various prompts can be used. Examples of prompts include those requesting a comparison of a first image and a second image, and those requesting a description of the visual transition from the first image to the second image. In some examples, a predetermined set of prompts is used to generate a natural language text description of the AR effect. Therefore, in some cases, the prompt is a natural language text prompt, and the description is a natural language text description.

[0037] In some examples, the method includes evaluating the visual semantic machine learning model's response to the corresponding cue and selecting only a subset of the outputs to include in the description of the AR effect. For example, as further described below, when generating the final description, only outputs determined to have sufficient relevance or similarity scores may be included or considered.

[0038] In some examples, descriptions of AR effects are stored in association with identifiers for the AR effects. AR effects can be indexed in the interactive system based on these descriptions. One approach may include receiving a search query from a user's device within the interactive system. The search query may be matched against an AR effect (e.g., based on a match between the search query and the description of an AR effect indexed in the interactive system). In response to an AR effect matching the search query and / or a user selection of an AR effect, the interactive system may cause image data with the applied AR effect to be presented at the user's device.

[0039] This method may include determining the category of an AR effect based on a description generated for that AR effect. The category may be stored in association with an identifier for the AR effect.

[0040] Visual semantic machine learning models can be used to generate descriptions of multiple different AR effects. Each description can be stored in a database that matches each AR effect with its corresponding description.

[0041] The examples described in this paper allow for the automatic and large-scale generation of natural language-based descriptions (e.g., searchable annotations or tags) of AR effects by leveraging visual question answering (VQA) or targeted image annotation / description capabilities of machine learning models. For example, these descriptions can be used to automatically index a large number (e.g., thousands) of AR effects within an interactive system. Improving the quality of the descriptions (e.g., by providing descriptions specifically focused on the enhanced visual effects based on "before and after" analysis) can improve the relevance or accuracy of search results.

[0042] The examples described in this article offer numerous technical advantages, such as facilitating the retrieval of AR effects from storage locations, improving the quality of search results in interactive applications, allowing for easier generation of insights related to similarities or differences between AR effects, and improving the rating, ranking, or classification of AR effect enhancements.

[0043] In some examples, the techniques described herein improve the functionality of content moderation systems by providing more accurate, detailed, or comprehensive descriptions of AR effects in an automated manner. For instance, by applying the automated AR effect description generation methods described herein, computer systems can be improved to accurately identify AR effects (e.g., to propose new AR effects for use in interactive systems) that include prohibited or offensive content.

[0044] When taking into account the effects described in this disclosure, one or more of the methods described herein can eliminate the need for certain work or resources that would otherwise be involved in generating or indexing AR effect descriptions or retrieving related AR effects. For example, by providing more accurate or comprehensive AR effect search results or recommendations, the computational resources used by one or more machines, databases, or networks can be utilized more efficiently or even reduced, thereby reducing the need for duplicate searches or further filtering of user searches. Examples of such computational resources may include processor cycles, network traffic, memory usage, graphics processing unit (GPU) resources, data storage capacity, power consumption, and cooling capacity.

[0045] Networked computing environment

[0046] Figure 1 This is a block diagram illustrating an example interactive system 100 for facilitating interactions over a network, such as exchanging text messages, making text, audio, and video calls, or playing games. The interactive system 100 includes multiple user systems 102, each of which hosts multiple applications including an interactive client 104 (as an example of an interactive application) and other applications 106. Each interactive client 104 is communicatively coupled to (e.g., hosted on corresponding other user systems 102) other instances of the interactive client 104, an interactive server system 110, and a third-party server 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 may also communicate with the locally hosted applications 106 using an application programming interface (API).

[0047] Each user system 102 may include multiple user devices, such as computing device 114, head-mounted wearable device 116, and computer client device 118, which are communicatively connected to exchange data and messages.

[0048] Interactive client 104 interacts with other interactive clients 104 and with interactive server system 110 via network 108. The data exchanged between interactive clients 104 (e.g., interaction 120) and between interactive client 104 and interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0049] Interactive server system 110 provides server-side functionality to interactive client 104 via network 108. While some functions of interactive system 100 are described herein as being performed by interactive client 104 or interactive server system 110, the location of certain functions—whether within interactive client 104 or interactive server system 110—may be a design choice. For example, it might be technically preferred that certain technologies and functions are initially deployed within interactive server system 110, but later migrated to interactive client 104, where user system 102 has sufficient processing power.

[0050] The interactive server system 110 supports various services and operations provided to the interactive client 104. Such operations include sending data to and receiving data from the interactive client 104, and processing data generated by the interactive client 104. This data may include message content, client device information, geolocation information, content enhancements (e.g., filters and overlays), message content persistence conditions, entity relationship information, and live event information. Data exchange within the interactive system 100 is activated and controlled via functions available through the user interface of the interactive client 104.

[0051] Specifically, turning to interactive server system 110, API server 122 is coupled to interactive server 124 and provides it with a programming interface, making the functionality of interactive server 124 accessible to interactive client 104, other applications 106, and third-party server 112. Interactive server 124 is communicatively coupled to database server 126, thereby facilitating access to database 128, which stores data associated with the interactions processed by interactive server 124. Similarly, web server 130 is coupled to interactive server 124 and provides a web-based interface to interactive server 124. To this end, web server 130 handles incoming network requests via Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0052] API server 122 receives and sends interactive data (e.g., command and message payloads) between interactive server 124 and user system 102 (and, for example, interactive client 104 and other applications 106) and third-party server 112. Specifically, API server 122 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by interactive client 104 and other applications 106 to activate the functionality of interactive server 124. API server 122 exposes various functions supported by interactive server 124, including account registration; login functionality; sending interactive data from one interactive client 104 to another interactive client 104 via interactive server 124; transferring media files (e.g., images or videos) from interactive client 104 to interactive server 124; setting media data sets (e.g., stories); retrieving the friend list of users of user system 102; retrieving messages and content; retrieving or applying AR effects; adding and deleting entities (e.g., friends) against an entity graph (e.g., entity graph 308); locating friends in the entity graph; and opening (e.g., application events associated with interactive client 104). Interactive server 124 hosts multiple systems and subsystems, as shown below. Figure 2 Describe it.

[0053] Application of links

[0054] Returning to interactive client 104, the features and functionality of external resources (e.g., linked application 106 or applet) can be made available to the user via the interface of interactive client 104. In this context, "external" refers to the fact that application 106 or applet is outside of interactive client 104. External resources are typically provided by third parties, but may also be provided by the creator or provider of interactive client 104. Interactive client 104 receives user selections regarding options to launch or access the features of such external resources. External resources may be application 106 installed on user system 102 (e.g., a "local application"), or a smaller version (e.g., a "applet") of an application hosted on user system 102 or remotely on user system 102 (e.g., on a third-party server 112). A smaller version of an application includes a subset of the features and functionality of the application (e.g., a full-scale, local version of the application) and is implemented using markup language documentation. In some examples, a smaller version of an application (e.g., a "applet") is a web-based markup language version of the application and is embedded in interactive client 104. In addition to using markup language documentation (e.g., In addition to files, mini-programs can also include scripting languages ​​(e.g., ... (file or .json file) and stylesheets (e.g.) document).

[0055] In response to receiving a user's selection of an option to launch or access an external resource, interactive client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, application 106, locally installed on user system 102, can be launched independently of and separately from interactive client 104, for example, by selecting the icon corresponding to application 106 on the home screen of user system 102. A smaller version of such an application can be launched or accessed via interactive client 104, and in some examples, no part of the smaller application can be accessed outside of interactive client 104, or only a limited portion of the smaller application can be accessed outside of interactive client 104. The smaller application can be launched by interactive client 104 by receiving, for example, markup language documents associated with the smaller application from third-party server 112 and processing such documents.

[0056] In response to determining that the external resource is a locally installed application 106, the interactive client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 104 communicates with a third-party server 112 (e.g.) to obtain a markup language document corresponding to the selected external resource. The interactive client 104 then processes the obtained markup language document to render the web-based external resource within the user interface of the interactive client 104.

[0057] Interactive client 104 can notify users of user system 102, or other users associated with such users (e.g., "friends"), of activities occurring in one or more external resources. For example, interactive client 104 can provide participants in a conversation (e.g., a chat session) within interactive client 104 with notifications related to the current or recent use of external resources by one or more members of a group of users. One or more users can be invited to join an active external resource or to activate (in a group of friends) a recently used but currently inactive external resource. External resources can provide participants in the conversation, each using the corresponding interactive client 104, with the ability to share items, conditions, states, or locations within the external resource with one or more members of a group of users during a chat session. Shared items can be interactive chat cards, which chat members can use to interact, for example, activate the corresponding external resource, view specific information within the external resource, or take chat members to a specific location or state within the external resource. Within a given external resource, response messages can be sent to the user on interactive client 104. External resources can selectively include different media items in the response based on the current context of the external resource.

[0058] Interactive client 104 can display a list of available external resources (e.g., application 106 or mini-program) to the user to launch or access a given external resource. This list can be presented in a context-sensitive menu. For example, icons representing different applications within application 106 (or mini-program) can vary depending on how the user launches the menu (e.g., from a conversational interface or a non-conversational interface).

[0059] System Architecture

[0060] Figure 2 This is a block diagram illustrating further details of an interactive system 100 according to some examples. Specifically, the interactive system 100 is shown as including an interactive client 104 and an interactive server 124. The interactive system 100 includes multiple subsystems supported on the client side by the interactive client 104 and on the server side by the interactive server 124. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) may have components that enable the microservice subsystem to operate independently and communicate with other services. Example components of a microservice subsystem may include:

[0061] Functional logic: Functional logic implements the functions of the microservice subsystem and represents the specific capabilities or functions provided by the microservice.

[0062] API Interfaces: Microservices can use lightweight protocols such as Representational State Transfer (REST) ​​or messaging to communicate with each other through well-defined APIs or interfaces. API interfaces define the inputs and outputs of a microservice subsystem and how it interacts with other microservice subsystems within the interactive system 100.

[0063] Data storage: The microservice subsystem can be responsible for its own data storage, which can be in the form of a database, cache, or other storage mechanisms (e.g., using database server 126 and database 128). This allows the microservice subsystem to operate independently of other microservices in the interactive system 100.

[0064] Service discovery: Microservice subsystems can find other microservice subsystems of the interactive system 100 and communicate with them. The service discovery mechanism enables microservice subsystems to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0065] Monitoring and logging: Microservice subsystems may need to be monitored and logged to ensure availability and performance. Monitoring and logging mechanisms enable the tracking of the health and performance of microservice subsystems.

[0066] In some examples, the interactive system 100 may employ a monolithic architecture, a service-oriented architecture (SOA), a functionality-as-a-service (FaaS) architecture, or a modular architecture. Example subsystems are... Figure 2 It is shown in the figure and discussed below.

[0067] Image processing system 202 provides various functions that enable users to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. Camera device system 204 includes (e.g., in a camera device application) control software that (e.g., directly or via an operating system) interacts with and controls the hardware camera device hardware of user system 102 to modify and enhance real-time images captured and displayed via interactive client 104.

[0068] Enhancement system 206 provides functionality related to the generation and distribution of enhancements, also known as AR effects (e.g., filters or media overlays), for images captured in real-time by the camera device of user system 102 or retrieved from the memory of user system 102. For example, enhancement system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) to interactive client 104 to enhance real-time images received via camera device system 204 or stored images retrieved from the memory of user system 102. These enhancements are selected by enhancement system 206 and presented to the user of interactive client 104 based on several inputs and data, such as:

[0069] The geographic location of user system 102; and

[0070] User entity relationship information of users in user system 102.

[0071] Enhanced or AR effects can include audio and visual content and visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at user system 102 for transmission in messages, or to video content such as video content streams or feeds sent from interactive client 104. Therefore, image processing system 202 can interact with and support various subsystems of communication system 208, such as messaging system 210 and video communication system 212.

[0072] Media overlays may include text or image data that can be superimposed on photographs taken by user system 102 or video streams produced by user system 102. In some examples, media overlays may be location overlays (e.g., Venice Beach), names of live events, or names of businesses (e.g., beach cafes). In other examples, image processing system 202 uses the geolocation of user system 102 to identify media overlays that include the names of businesses located at the geolocation of user system 102. Media overlays may include additional identifiers associated with the businesses. Media overlays may be stored in database 128 and retrieved via database server 126.

[0073] Image processing system 202 provides a user-based publishing platform that allows users to select a geographic location on a map and upload content associated with that location. Users can also specify which media overlays should be provided to other users. Image processing system 202 generates a media overlay that includes the uploaded content and associates it with the selected geographic location.

[0074] The enhancement creation system 214 supports an AR developer platform and includes applications for content creators (e.g., artists and developers) to create and publish enhancements (e.g., AR experiences) to interactive clients 104. The enhancement creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technologies, and templates. In some examples, the enhancement creation system 214 provides a merchant-based publishing platform that allows merchants to select specific enhancements associated with geolocation via a bidding process. For example, the enhancement creation system 214 associates the media overlay of the highest-bidding merchant with a corresponding geolocation for a predefined amount of time.

[0075] Communication system 208 is responsible for enabling and processing various forms of communication and interaction within interactive system 100, and includes messaging system 210, audio communication system 216, and video communication system 212. In some cases, messaging system 210 is responsible for enabling temporary or time-limited access to content by interactive client 104. Messaging system 210 includes (e.g., in a short-lived timer system) multiple timers that selectively enable access (e.g., for presentation and display) of messages and associated content via interactive client 104 based on duration and display parameters associated with a message or set of messages (e.g., a story). Audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 104.

[0076] User management system 218 is operationally responsible for managing user data and profiles, and maintaining information about users of interactive system 100 and the relationships between users (e.g., stored in...). Figure 3 The entity information shown in Entity Table 306, Entity Diagram 308 and Profile Data 302.

[0077] The collection management system 220 is operationally responsible for managing collections or sets of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 220 can also be responsible for publishing icons that notify the user interface of the interactive client 104 of the availability of specific collections. The collection management system 220 includes curation functions that enable collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 220 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users may be compensated for including user-generated content in a collection. In such cases, the collection management system 220 operates to automatically pay such users for using their content.

[0078] Map system 222 provides various geolocation functions and supports the presentation of map-based media content and messages by interactive client 104. For example, map system 222 enables the display (e.g., stored in profile data 302) of user icons or avatars on the map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content generated by these friends (e.g., a collection of messages including photos and videos). For example, on the map interface of interactive client 104, messages posted by a user from a specific geolocation to interactive system 100 can be displayed to the specific user's "friends" within the context of that specific location on the map. Users can also share their location and status information with other users of interactive system 100 via interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of the map interface of interactive client 104.

[0079] Game system 224 provides various game functions within the context of interactive client 104. Interactive client 104 provides a game interface that offers a list of available games that can be initiated by a user within the context of interactive client 104 and played with other users of interactive system 100. Interactive system 100 also enables specific users to invite other users to participate in specific games by sending invitations from interactive client 104. Interactive client 104 also supports sending and receiving voice, video, and text messages (e.g., chat) within the context of playing the game, provides leaderboards for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0080] External resource system 226 provides interactive client 104 with an interface to communicate with remote servers (e.g., third-party server 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts applications or smaller versions of applications (e.g., game applications, utility applications, payment applications, or ride-sharing applications) based on markup languages ​​(e.g., HTML5). Interactive client 104 can launch web-based resources (e.g., applications) by accessing HTML5 files from the third-party server 112 associated with the web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript using a software development kit (SDK) provided by interactive server 124. The SDK includes APIs with functionality that can be called or activated by the web-based application. Interactive server 124 hosts a JavaScript library that provides access to a given external resource for specific user data of interactive client 104. HTML5 is an example of a technology used for programming games, but applications and resources programmed using other technologies can be used.

[0081] To integrate the SDK's functionality into the web-based resource, the third-party server 112 downloads the SDK from the interactive server 124, or the third-party server 112 otherwise receives the SDK. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interactive client 104 into the web-based resource.

[0082] The SDK stored on the interactive server system 110 effectively bridges the gap between external resources (e.g., application 106 or applet) and the interactive client 104. This provides users with a seamless experience communicating with other users on the interactive client 104 while preserving the look and feel of the interactive client 104. To bridge communication between the external resources and the interactive client 104, the SDK facilitates communication between a third-party server 112 and the interactive client 104. A bridging script running on the user system 102 establishes two unidirectional communication channels between the external resources and the interactive client 104. Messages are sent asynchronously between the external resources and the interactive client 104 via these communication channels. Each SDK function activation is sent as a message and callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with that callback identifier.

[0083] By using the SDK, not all information from the interactive client 104 is shared with the third-party server 112. The SDK limits which information is shared based on the needs of the external resources. Each third-party server 112 provides the interactive server 124 with an HTML5 file corresponding to the web-based external resource. The interactive server 124 can add a visual representation (e.g., box design or other graphics) of the web-based external resource to the interactive client 104. Once the user selects the visual representation or instructs the interactive client 104 to access the features of the web-based external resource through the graphical user interface of the interactive client 104, the interactive client 104 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0084] Interactive client 104 presents a graphical user interface (GUI) for an external resource (e.g., a login page or title screen). During, before, or after presenting the login page or title screen, interactive client 104 determines whether the initiated external resource has previously been authorized to access user data of interactive client 104. In response to determining that the initiated external resource has previously been authorized to access user data of interactive client 104, interactive client 104 presents another GUI of the external resource, including its functionality and characteristics. In response to determining that the initiated external resource has not previously been authorized to access user data of interactive client 104, after displaying the login page or title screen of the external resource for a threshold time period (e.g., 3 seconds), interactive client 104 slides up a menu (e.g., animates the menu to appear from the bottom of the screen to the middle or other parts of the screen) to authorize the external resource to access user data. This menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, interactive client 104 adds the external resource to the list of authorized external resources and allows the external resource to access user data from interactive client 104. External resources are authorized by the interactive client 104 to access user data under the OAuth 2 framework.

[0085] Interactive client 104 controls the type of user data shared with external resources based on the type of authorized external resource. For example, it provides access to a first type of user data (e.g., two-dimensional avatars of users with or without different avatar characteristics) to external resources including full-scale applications (e.g., application 106). As another example, it provides access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar characteristics) to external resources including smaller versions of applications (e.g., web-based versions of applications). Avatar characteristics include different ways of customizing the appearance and feel of an avatar (e.g., different poses, facial features, clothing, etc.).

[0086] The advertising system 228 enables third parties to purchase advertisements to be presented to end users via the interactive client 104, and also handles the delivery and presentation of these advertisements.

[0087] Artificial intelligence and machine learning system 230 provides various services to different subsystems within interactive system 100. For example, artificial intelligence and machine learning system 230 operates in conjunction with image processing system 202 and camera device system 204 to analyze images and extract information, such as objects, text, or faces. This information can then be used by image processing system 202 to enhance, filter, or manipulate (e.g., apply AR effects) the image.

[0088] As described below, the artificial intelligence and machine learning system 230 can also operate in conjunction with the AR effect description system 232 to analyze images and generate descriptions of AR effects that are applied to or presented in these images.

[0089] Artificial intelligence and machine learning systems 230 can implement visual semantic machine learning models (also known as visual language models). As used herein, the term "visual semantic machine learning model" refers to a machine learning model or a combination of machine learning models that has the ability to process both visual (e.g., images or videos) and linguistic (e.g., text) data. Visual semantic machine learning models combine visual and semantic modalities, providing applications such as image annotation and VQA. Examples of the use of visual semantic machine learning models are described below:

[0090] Image annotation: Visual semantic machine learning models receive input images and generate natural language text descriptions or annotations for the images.

[0091] VQA: A visual semantic machine learning model receives an image and a prompt (e.g., a question related to an image in natural language text format) and generates a natural language text response to the prompt based on the image. In the context of this disclosure, the word "problem" in "visual question answering" should be interpreted broadly, as "problem" can be a closed-ended question or a more open-ended prompt (e.g., a request to describe an image).

[0092] Text-to-image synthesis: Visual semantic machine learning models receive a textual description of a scene or object and generate an image designed to match that description.

[0093] Semantic image retrieval: Visual semantic machine learning models receive text queries and retrieve one or more images that are considered to match (or be most relevant to) the text query based on semantic understanding.

[0094] Image-to-text translation: Visual semantic machine learning models receive images containing text (e.g., road signs or pages in a book), extract the text, and translate the text into another (natural) language.

[0095] Phrase localization: Visual semantic machine learning models perform object detection from input images and natural language phrases.

[0096] Examples of machine learning models that can provide such multimodal capabilities include models based on "BLIP" (Bootstaff Language Image Pre-training) strategies (e.g., BLIP and BLIP-2). For example, BLIP-2 is a scalable multimodal pre-training method that enables Large Language Models (LLMs) to ingest and understand images, thereby achieving image-to-text generation, VQA, and other functionalities. Another example of a machine learning model is a CLIP (Contrastive Language Image Pre-training) type model. More details about example machine learning models are provided below.

[0097] Artificial intelligence and machine learning system 230 can be used by augmentation system 206 to generate augmented content and AR experiences, such as adding virtual objects or animations to real-world images. Communication system 208 and messaging system 210 can use artificial intelligence and machine learning system 230 to analyze communication patterns and provide insights into how users interact with each other, as well as provide intelligent message classification and tagging, such as classifying messages based on sentiment or topic.

[0098] The artificial intelligence and machine learning system 230 can also provide chatbot functionality for message interactions 120 between user systems 102 and between user systems 102 and the interaction server system 110. The artificial intelligence and machine learning system 230 can provide various generative functions (e.g., allowing users to generate text, images, or video content based on prompts). The artificial intelligence and machine learning system 230 can work with the audio communication system 216 to provide speech recognition and natural language processing capabilities, enabling users to interact with the interaction system 100 using voice commands.

[0099] Referring now to AR effect description system 232, as described above, it is expected that descriptions of AR effects will be generated. AR effect description system 232 can perform various functions related to the generation and / or storage of descriptions of AR effects. For example, in the case where interactive system 100 supports a large number of AR effects (e.g., thousands or even millions of AR effects), AR effect description system 232 can implement an automated pipeline (e.g., by working with image processing system 202) to render each of multiple AR effects on appropriate content items, operate in conjunction with artificial intelligence and machine learning system 230 to generate descriptions of AR effects, and store these descriptions for downstream use in the context of interactive system 100. In some examples, the description or portions thereof (e.g., words or phrases) can be tokenized for downstream use (e.g., for input into an embedding model).

[0100] Data Architecture

[0101] Figure 3This is a schematic diagram illustrating a data structure 300 that can be stored in a database such as database 128 of an interactive server system 110, according to certain examples. Although the contents of database 128 are shown as including multiple tables, it should be understood that data can be stored in other types of data structures, such as object-oriented databases.

[0102] Database 128 includes message data stored in message table 304. For any given message, this message data includes at least message sender data, message receiver (or recipient) data, and payload. See below for reference. Figure 13 Further details are provided regarding information that can be included in the message and is contained within the message data stored in message table 304.

[0103] Entity table 306 stores entity data and (for example, links to entity diagram 308 and profile data 302). Entities for which records are maintained in entity table 306 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of the entity type, any entity for which the interactive server system 110 stores data can be an identifiable entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0104] Entity Graph 308 stores information about relationships and associations between entities. As an example only, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based. Some relationships between entities can be one-way, such as an individual user subscribing to digital content (e.g., a newspaper or other digital media channel or brand) for a business or publishing user. Other relationships can be two-way, such as the "friend" relationship between the various users of Interactive System 100.

[0105] Certain permissions and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friend relationship between individual users) can include authorization for the public disclosure of digital content items between the individual users, but certain restrictions or filters can be imposed on the public disclosure of these digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual user and a business user can impose varying degrees of restrictions on the public disclosure of digital content from the business user to the individual user, and can significantly limit or prevent the public disclosure of digital content from the individual user to the business user. As an example of an entity, a specific user can record certain restrictions in the records for that entity within entity table 306 (e.g., through privacy settings). Such privacy settings can be applied to all types of relationships in the context of the interaction system 100, or selectively applied to certain types of relationships.

[0106] Profile data 302 stores various types of profile data about a specific entity. Based on privacy settings specified by the specific entity, profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case of an individual, profile data 302 includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a set of such avatar representations) selected by the user. The specific user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interaction system 100 and on a map interface displayed to other users by the interaction client 104. The set of avatar representations may include “status avatars,” which present a graphical representation of a status or activity that the user can choose to transmit at a specific time.

[0107] In the case that the entity is a group, in addition to the group name, members and various settings for the relevant group (e.g., notifications), the profile data 302 for the group may similarly include one or more avatars associated with the group.

[0108] Database 128 also stores augmentation data in augmentation table 310, including AR effects such as filters and overlays. The augmentation data is associated with and applied to videos (video data is stored in video table 312) and images (image data is stored in image table 314).

[0109] In some examples, filters include visual content displayed as an overlay on an image or video during presentation to the recipient user. Filters can be of various types, including user-selected filters from a set of filters presented to the sending user by the interactive client 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geofilters), which can be presented to the sending user based on geolocation. For example, geolocation filters specific to nearby or particular locations can be presented by the interactive client 104 within the user interface based on geolocation information determined by the GPS unit of the user system 102.

[0110] Another type of filter is a data filter, which can be selectively presented to the sending user by the interactive client 104 based on other inputs or information collected by the user system 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed of the sending user, the battery life of the user system 102, or the current time.

[0111] Other augmented data that can be stored within image table 314 includes, for example, AR content items corresponding to the application's "Lens" or AR experience. AR content items can include real-time effects that can be added to images or videos.

[0112] Collection table 316 stores data about collections of messages and associated image, video, or audio data, compiled into collections (e.g., stories or galleries). The creation of a specific collection can be initiated by a specific user (e.g., each user whose records are maintained in entity table 306). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. For this purpose, the user interface of interactive client 104 can include user-selectable icons to allow the sending user to add specific content to their personal story.

[0113] The collection can also constitute a "live story," which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic technologies. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a co-located event at a specific time can be presented with the option to contribute content to a specific live story, for example, via the user interface of interactive client 104. Live stories can be identified by interactive client 104 based on a user's location. The end result is a "live story" told from a community perspective.

[0114] Another type of content collection is called a "location story," which allows users of user system 102 located in a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some examples, contributions to a location story may employ secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0115] As mentioned above, video table 312 stores video data, which in some examples is associated with messages for which records are maintained within message table 304. Similarly, image table 314 stores image data associated with messages whose message data is stored in entity table 306. Entity table 306 can associate various enhancements from enhancement table 310 with various images and videos stored in image table 314 and video table 312.

[0116] AR effect description table 318 stores descriptions of various AR effects supported by interactive system 100. For example, each AR effect may have a unique identifier (e.g., code or name), where a natural language text description of the AR effect is stored in association with the identifier in AR effect description table 318. The descriptions facilitate searching. For example, when a user of user system 102 searches for keywords via interactive client 104, interactive client 104 can locate matching or related AR effects by examining the descriptions in AR effect description table 318. In some cases, the data in AR effect description table 318 can be indexed for easy searching and retrieval.

[0117] In some examples, the AR effect description system 232 of the interactive system 100 automatically generates descriptions of AR effects (e.g., new AR effects generated by users of the augmentation creation system 214 and uploaded to the interactive system 100) and stores these descriptions in the AR effect description table 318 for downstream use. The AR effect description table 318 may also include other AR effect metadata, such as AR effect tags, categories, groups, or rankings. For example, the AR effect description system 232 may use a visual semantic machine learning model to generate AR effect descriptions and also identify the category of the AR effect (e.g., sports, birthday, or dog), and then store the category in the AR effect description table 318 in association with the AR effect's identifier.

[0118] Cue table 320 stores multiple cue items. One or more cue items can be used in the automatic generation of descriptions for AR effects by feeding them together with or in association with one or more images into a visual semantic machine learning model to obtain output describing the AR effect.

[0119] Figure 4 This is shown based on some examples. Figure 1 and Figure 2 Figure 400 shows the components of the AR effect description system 232 of the interactive system 100. The AR effect description system 232 communicates with the database 128 (e.g., for data storage and retrieval) and with the artificial intelligence and machine learning system 230 to utilize machine learning capabilities.

[0120] In some examples, Figure 4At least some of the components shown are configured to communicate with each other to implement the aspects described herein. One or more of the components described herein may be implemented using hardware (e.g., one or more processors of one or more machines) or a combination of hardware and software. For example, a component described herein may be implemented by a processor configured to perform the operations described herein for that component. Furthermore, two or more of these components may be combined into a single component, or the functionality described herein for a single component may be further divided among multiple components. Additionally, according to various examples, the components described herein may be implemented using a single machine, database, or device, or may be distributed across multiple machines, databases, or devices.

[0121] AR effect description system 232 is shown as including rendering component 402, video collection component 404, frame extraction component 406, stitching component 408, prompt component 410, description generation component 412 and data storage component 414.

[0122] Rendering component 402 is responsible for rendering AR effects on content items such as images or videos. For example, rendering component 402 can access a first video or base video that does not include a certain AR effect, and then apply the AR effect to the base video to produce a second video or enhanced video. For example, the first video could be a video of a person waving at the camera of a recording device, and the AR effect could be a color filter that changes the color of the person's hair (e.g., from brown to blue), so that the enhanced video shows the person waving at the camera but with a different hair color than in the base video. Rendering component 402 can similarly apply AR effects to still images to produce enhanced images.

[0123] The rendering component 402 can work with the image processing system 202, for example, to perform object detection and apply AR effect algorithms to content items. The rendering component 402 can store enhanced content items (e.g., stored in database 128 along with corresponding basic content items).

[0124] Rendering component 402 can implement or access automated rendering services (e.g., automated AR effect rendering services), which may be referred to as "lens rendering services" or "filter rendering services." Such services can provide a pipeline of suitable content, including enhanced video or enhanced images. In some examples, the service includes an application configured to render specific AR effects on a base video (which may also be referred to as a substrate video). In other words, the rendering service provides an automated service capable of supplying video or image content on which desired AR effects can be applied. The rendering service can retrieve videos from a library in database 128. For example, the rendering service can access the base video and individually render (by way of example only) 50, 100, or 1200 AR effects onto the base video to obtain 50, 100, or 1200 enhanced videos for downstream use.

[0125] The video collection component 404 can be configured to (e.g., from database 128) retrieve or collect pairs of videos. As described above, a pair of videos can include a base video and its corresponding enhanced video.

[0126] Frame extraction component 406 is responsible for extracting corresponding frames from the base video and its associated enhanced video. Frame extraction component 406 can be configured to extract individual frames from digital video files or streams.

[0127] For example, and as mentioned above, the base video could depict a person waving at the camera, and its associated enhanced video could depict a person with different hair colors produced by AR effects. The frame extraction component 406 can preprocess the video by extracting corresponding frames from the base video and the enhanced video to allow for useful comparisons of the visual features therein.

[0128] Corresponding frames can be frames with the same (or substantially the same) timestamp or the same (or substantially the same) sequence position between the base video and the augmented video. In this way, objects, environment, and positions in the two extracted frames can be substantially aligned, except for AR effects (which are not present in the base video) and changes caused by AR effects. The frame extracted from the base video is referred to below as the "first image," and the frame extracted from the augmented video is referred to below as the "second image."

[0129] In some examples, frames are selected based on the differences between the base video and the enhanced video. For instance, frame extraction component 406 can automatically select a pair of frames (e.g., a first image and a second image) that includes a frame from the base video and a temporally corresponding frame from the enhanced video, where there is a high degree of pixel difference between the two frames. In some cases, the pair of frames that provides the highest degree of pixel difference can be selected.

[0130] To determine the difference between a pair of frames (e.g., a first image and a second image), a pixel-wise loss between the frames can be calculated. In this way, the frame extraction component 406 can be able to evaluate multiple pairs and find a frame index that provides a contrasting image useful in representing related AR effects (or aspects thereof).

[0131] In some examples, the first and second images are stitched or combined before they are processed for AR effect description generation. The stitching component 408 can access the two corresponding images and stitch them together, for example, based on a predetermined spatial arrangement. For example, the first and second images can be joined horizontally so that they are positioned side by side in the stitched image, or vertically so that one image is on top of the other.

[0132] Designing and / or implementing systems or components capable of understanding AR effects (e.g., media overlays, filters, or AR experiences) can be a challenging task. As mentioned above, to obtain a useful description of the AR effect, a first and second image, along with one or more suitable cues, can be fed into a visual semantic machine learning model. In response, the visual semantic machine learning model can provide outputs describing one or more features of the AR effect. The cue component 410 can be configured to send the image and cue (e.g., a natural language text cue from cue table 320) to a visual semantic machine learning model 416 implemented by the artificial intelligence and machine learning system 230, and receive output (e.g., a natural language text response) from the visual semantic machine learning model 416.

[0133] In some examples, the hint component 410 implements VQA and / or directional image annotations. Examples of hints are provided below.

[0134] The description generation component 412 is responsible for generating, selecting, aggregating, enhancing, or modifying descriptions for specific AR effects. In some cases, the description generation component 412 may select outputs from the visual semantic machine learning model 416 and directly use or apply such outputs as descriptions of the relevant AR effects. In other cases, the description generation component 412 may generate new or modified descriptions based on the outputs from the visual semantic machine learning model 416. In some examples, the description generation component 412 uses only a subset of the outputs generated by the visual semantic machine learning model 416 (e.g., by ignoring outputs identified as irrelevant or potentially irrelevant).

[0135] For example, the visual semantic machine learning model 416 can generate multiple outputs describing different features of the AR effect, such as a first output indicating that the AR effect is a color-changing effect and a second output indicating that a specific effect is turning a person's hair blue. The description generation component 412 can automatically aggregate the first and second outputs to produce a final description that encapsulates the outputs received from the visual semantic machine learning model 416. In some examples, as further described below, the description generation component 412 (e.g., using a machine learning model such as the CLIP model) scores the multiple outputs and selects a subset of the outputs based on the scores.

[0136] Data storage component 414 can (e.g., in such a case) Figure 3 The AR effect description table 318 in the database 128 (shown) stores data related to AR effects. For example, the data storage component 414 can store the output or response received from the visual semantic machine learning model 416 in the database 128. The data storage component 414 can store the description of the AR effect selected, confirmed, or generated by the description generation component 412 in association with the identifier of the AR effect. The data storage component 414 can store such data for subsequent retrieval. For example, the data storage component 414 can store data in a format that facilitates retrieval in response to an AR effect search query originating from an interactive client 104 operating within the interactive system 100. Figure 3 The AR effect description table 318 is as follows (see below). Figure 9 and Figure 10 As described.

[0137] Figure 5 This is a flowchart illustrating a method 500 suitable for generating AR effects, based on some examples. The operations in method 500 can be used by the interactive system 100 as described above. Figures 1 to 4 The components described (e.g., parts, modules, systems, or engines) are used for execution. Therefore, reference is made to the relevant information through examples rather than limitations. Figures 1 to 4 The components and interaction system 100 are described to describe method 500. (Refer to...) Figures 6 to 8 The example image shown further illustrates method 500.

[0138] Note that, although the interactive system 100 is described... Figure 5 The operation refers to the actions performed by the components described herein, but it should be understood that at least some of these actions may be deployed on various other hardware configurations or performed by similar components residing elsewhere. For ease of reference, the term "operation" is used to refer to an element in the accompanying drawings of this disclosure, and it will be understood that each "operation" may identify one or more operations, processes, actions, or steps, and may be performed by one or more components.

[0139] Method 500 begins at the start of the loop element 502 and proceeds to operation 504, where the AR effect is applied to the base video to obtain an enhanced video. Figure 5 China (and) Figure 7 and Figure 8 (Let's observe together) Using the "dog face" filter as a non-limiting example of an AR effect. For example, the base video could be a video depicting a person, including their face, but without any AR effects applied to it. Then, rendering component 402 applies AR effects to add dog features to the person's face (see discussion below). Figure 7 ).

[0140] At operation 506, video collection component 404 retrieves the base video and enhanced video, and frame extraction component 406 extracts a pair of corresponding frames (e.g., temporally corresponding frames as discussed above). Thus, a first image and a second image are obtained. As mentioned, frame extraction component 406 may select the first image and the second image (instead of other image pairs from the base video and enhanced video) based on detected differences or contrasts (e.g., high pixel dissimilarity).

[0141] For example, a first image is from a base video and shows a person's "normal" (e.g., unenhanced) face, while a second image is from an enhanced video and shows the same content as the first image, except for dog features superimposed on the person's face by rendering component 402. Rendering component 402 can operate in conjunction with image processing system 202 to process the base video such that the enhanced video includes a "dog face" filter.

[0142] Figure 6 An example of the first image 600 is shown, and Figure 7 An example of the second image 700 is shown, which includes an AR effect 702 in the form of an example "dog face" filter. The first image 600 and the second image 700 are used below as examples to describe certain aspects of method 500. However, it should be understood that many other types of AR effects or images can be used in other examples of the technique described herein.

[0143] Method 500 proceeds to operation 508, where the stitching component 408 generates a stitched image including the first image 600 and the second image 700. Figure 8 An example of a stitched image 800 is shown, in which the first image 600 and the second image 700 are joined together in a horizontal stitching manner.

[0144] Specifically, in Figure 8In the stitched image 800, the first image 600 is located to the left of the stitched image 800, and the second image 700 is located to the right of the stitched image 800 to provide a left-to-right transformation 802 (as will be further described below). The stitched image 800 is a non-limiting example used to describe certain aspects of method 500, and it will be understood that other types of stitching (e.g., vertical stitching) may be used in other examples of the technique described herein. Furthermore, in some examples, the stitched image may include more than two images, such as a series of three frames from a base video and a corresponding (e.g., temporally corresponding) series of three frames from an enhanced video.

[0145] Now refer to Figure 5 The operation 510 then proceeds, for example, by prompting component 410, to provide the stitched image 800 along with a predetermined set of prompts to the visual semantic machine learning model 416. Various prompts can be utilized depending on the technical requirements or use case. Note that the prompts provided by prompting component 410 do not necessarily need to be “questions.” For example, a prompt could be a more open-ended request describing the visual effects or a question about the visual effects. In some examples, a combination of direct or closed-ended “questions” and more open-ended prompts may be used.

[0146] The prompt may request the visual semantic machine learning model 416 to compare the first image 600 and the second image 700, for example, by describing the visual transition from the first image 600 to the second image 700. This is done in a configuration that provides a left-to-right transformation 802. Figure 8 In the case of the stitched image 800, the visual semantic machine learning model 416 can be specifically prompted to analyze the differences between the left and right images, or the changes observed when transitioning across images from left to right. Multiple prompts can be automatically fed to the visual semantic machine learning model 416 together with or in association with the stitched image 800 to cover different aspects or features of the AR effect 702.

[0147] Some cues may be designed to elicit general or broad responses, such as a description of the overall effect or change produced by AR effect 702. Other cues may be designed to elicit more granular or specific responses, such as a "yes or no" question asking whether certain filters were applied or whether a color change was observed. In some cases, cues may be created that allow the visual semantic machine learning model 416 to describe the "before and after" effects of the relevant enhancement.

[0148] Depending on the implementation and specific architecture of the model with which the prompting component 410 communicates, prompts can be provided to the visual semantic machine learning model 416 individually or simultaneously. In some examples, the prompts used by the prompting component 410 are stored in... Figure 3 The prompt is in table 320.

[0149] Table 1 below includes non-restrictive examples of tips that can be used in some examples (in some cases only a subset may be used), while Table 2 includes tips specific to... Figure 8 Four example hints and example answers for stitching together an 800-pixel image.

[0150]

[0151] Table 1: Example Tips

[0152]

[0153] Table 2: and Figure 8 Related sample tips with example responses

[0154] Therefore, while a single cue may be used to generate a description of an AR effect in some examples, multiple cuees are utilized in others. Different cuees can provide outputs describing different dimensions, aspects, or characteristics of the AR effect, thereby improving the accuracy, relevance, or comprehensiveness of the final description.

[0155] Refer again Figure 5 In response to providing a cue to the visual semantic machine learning model 416, at operation 512, the cue component 410 receives the output from the visual semantic machine learning model 416. For example, in the case of stitching image 800 and the cue in Table 2, the cue component 410 may receive four responses as shown in Table 2 describing the features of the AR effect (e.g., the AR effect adds features of a dog to the face and the AR effect does not apply a global color filter).

[0156] By feeding appropriate cues to the visual semantic machine learning model 416, the model can automatically perform visual feature extraction and present natural language descriptions of these features for downstream use. For example, see below. Figure 9 and Figure 10 Describe downstream usage in the search in more detail.

[0157] By comparing two images (one image including AR effect 702 and the other image not including AR effect 702) using the visual semantic machine learning model 416, the visual semantic machine learning model 416 can be used to better separate relevant visual effects from other content (such as background content or objects unrelated to AR effect 702), thereby generating useful output.

[0158] In this way, the question or request answering capability of the visual semantic machine learning model 416 can be used to automate and / or improve the generation of AR effect descriptions. The output from the visual semantic machine learning model 416 can be stored (e.g., stored in...). Figure 3The AR effect description is in Table 318 or in the table storing the "original" or "preliminary" description.

[0159] exist Figure 5 In operation 514 of method 500, the description of the AR effect generated by the generation component 412 is described. As mentioned, the description generation component 412 can directly use the output of the visual semantic machine learning model 416 as the description, or it can modify the output.

[0160] In some examples, the description generation component 412 aggregates different responses into a single description of the AR effect, such as a single description of various aspects of the AR experience. In some examples, the description generation component 412 may leverage the machine learning model of the artificial intelligence and machine learning system 230 to summarize, filter, and / or adjust the output of the visual semantic machine learning model 416, or generate a set of keywords or tags describing the AR effect based on the output of the visual semantic machine learning model 416.

[0161] In some examples, the description generation component 412 can automatically evaluate the quality or relevance of the response to determine whether to include the response in the final description or whether to consider the response when generating the final description. CLIP is an example of a neural network model that can be used for this purpose. Given an image and a text description (e.g., an augmented video frame and a response generated by the visual semantic machine learning model 416), a CLIP-based model can determine how closely they match by computing a similarity score in a shared embedding space. The resulting score can be viewed as a measure of the relevance between a given image and text.

[0162] Therefore, the description generation component 412 can generate a relevance or similarity score, such as a CLIP score, for one (or each) output. A high score may indicate that the output is relevant and should be included in the final description or considered when generating the final description. Conversely, a low score may indicate a poor match or irrelevance. The description generation component 412 may, for example, include or consider all outputs with scores above a certain threshold, or include or consider a predetermined number of outputs with the highest scores.

[0163] In some examples, the description of generating component 412 utilizes an affinity score, which can be calculated as follows:

[0164] A text encoder using the CLIP model encodes relevant responses into embedding vectors.

[0165] The image encoder using the CLIP model also encodes the relevant images (e.g., augmented frames) into embedding vectors.

[0166] Then, the cosine similarity between the text embedding and the image embedding is calculated to obtain a similarity score (e.g., between -1 and 1).

[0167] The cosine similarity is an "affinity score" that reflects the degree of matching between text and image content.

[0168] Then, by taking, for example, max(0,100) cos_sim(text_embedding, image_embedding)) limits the score to between 0 and 100.

[0169] In a scenario where a visual semantic machine learning model 416 can be prompted with a predetermined number of cues (e.g., 20 different cues), only a subset of the output (e.g., the 10 highest-rated responses) can be utilized downstream. In this way, a diverse set of cues can be effectively used to generate useful and relevant descriptions.

[0170] Once a description is generated, data storage component 414 stores the description in association with the AR effect (e.g., the identifier of the AR effect in AR effect description table 318) (operation 516). This allows interactive system 100 to automatically generate and store descriptions indicating "what" each AR effect is about. In this way, the descriptions can be used for downstream purposes, such as for search or ranking purposes. In some examples, the descriptions of AR effects are used to index the data in AR effect description table 318 to facilitate searching and retrieval.

[0171] For example, users can use Figure 1 Mobile device 114 (which performs interactive client 104) to search Figure 7 The AR effect 702 (“dog face” filter) is applied. The user can input a search query (e.g., “dog face,” “dog tongue,” etc.) through the interactive client 104, and then match the search query against the description of AR effect 702 stored in the AR effect description table 318. In response to the AR effect 702 matching the search query, the interactive client 104 can cause image data to be presented at the mobile device 114 and apply AR effect 702 to the image data. For example, at operation 518, AR effect 702 can be applied to the image data. In some cases, AR effect 702 is applied in real time (e.g., applied to objects presented in the camera feed interface of the interactive client 104 presented on the mobile device 114, thereby allowing the user to capture images or videos including AR effect 702). In other cases, AR effect 702 can be applied to previously captured content loaded from memory. Method 500 ends at the end loop element 520.

[0172] Note that, as provided via horizontal splicing Figure 8 The left-to-right transformation 802 is a non-limiting example. For example, a right-to-left transformation can be provided by stitching images together, such that the first (e.g., before enhancement) image is on the right and the second (e.g., after enhancement) image is on the left. A top-to-bottom or bottom-to-top transformation can be provided via vertical stitching. In each case, a cue can be specifically designed or selected to guide the visual semantic machine learning model 416. For example, in the case of a top-to-bottom transformation, the cue can make it clear to the visual semantic machine learning model 416 that it should "look" for the AR effect in the bottom image or the bottom portion of the stitched image.

[0173] Although Figure 5 Method 500 involves stitching images (e.g., stitching image 800), but in other examples, images may be fed separately rather than in a stitched form to a machine learning model (e.g., visual semantic machine learning model 416). For example, in some cases, a first image and a second image may be provided separately to the visual semantic machine learning model, which is then prompted to compare the two images and describe the observed visual effects.

[0174] As mentioned above, in some cases, it may be desirable to use more than two images. For example, in the case of a specific AR effect (e.g., an AR experience), different visual effects may be observed at different times (e.g., different stages of the AR experience). In this case, multiple pairs of images can be extracted from the base video and the augmented video separately.

[0175] For example, Figure 4 The frame extraction component 406 can extract a first image pair corresponding to a first time point in the two videos, a second image pair corresponding to a second time point in the two videos, and a third image pair corresponding to a third time point in the two videos, wherein each image pair shows different or varying features of the AR effect. A visual semantic machine learning model (e.g., visual semantic machine learning model 416) can then be trained to describe the visual effects observed at these different stages, thereby allowing the generation of a comprehensive description of the AR effect.

[0176] In some examples, the techniques described in this paper can improve the functionality of content moderation systems by providing more accurate, detailed, or comprehensive descriptions of AR effects in an automated manner. Content moderation can be simplified, for example, by reducing the need for manual review of AR effects.

[0177] For example, return to reference Figure 2The enhanced creation system 214 enables developers or content creators to create new AR effects and publish them to the interactive system 100. However, it may be desirable to review such new AR effects for offensive or harmful content. Appropriate prompts (such as prompt number 11 or prompt number 15 in Table 1) can be used to help identify such content. In some cases, the AR effect description system 232 can automatically determine, based on the description of the AR effect, that the AR effect contains or produces offensive or harmful content, and accordingly mark the AR effect in the AR effect description table 318. In response to detecting this mark, the interactive system 100 may then prevent the AR effect from being used on the interactive client 104, or may automatically impose usage restrictions (e.g., age restrictions).

[0178] As described elsewhere, once an AR effect description is generated, the data storage component 414 can store the description associated with the AR effect. Table 3 below includes examples of such descriptions, which may be generated and / or aggregated, for example, using the techniques described herein, and stored in AR effect description table 318 for use in the interactive system 100. The example provided is a panda-related AR effect and is provided for illustrative purposes only.

[0179]

[0180] Table 3: Examples of AR Effect Descriptions

[0181] For example, referring to the AR effect identified by AR effect ID 23101 in Table 3, this AR effect is described as adding pandas and leaves to an image of a person (using the automation techniques described herein). Based on some examples, Figure 9 The first image 900 is shown, and Figure 10 A second image 1000 is shown, including an AR effect 1002 identified by ID 23101. The first image 900 may be from a base video and shows a "normally captured" (e.g., unenhanced) image of a person, while the second image 1000 may be from an enhanced video and show the same content as the first image 900 but with a panda-related AR effect 1002 applied (e.g., as applied by rendering component 402).

[0182] Users can use Figure 1The mobile device 114 (which executes the interactive client 104) searches for panda-related AR effects. For example, a user can enter the following search query: "Panda in a bamboo forest." The interactive client 104 can then match the search query with the most relevant descriptions in the AR effect description table 318 (e.g., one or more descriptions in Table 3). The interactive client 104 can identify the AR effect ID associated with each relevant description to display the relevant AR effect.

[0183] In some examples, the interactive client 104 can obtain a ranking of relevant AR effects, for example, based on the degree of matching between a search query and the corresponding description in the AR effect description table 318 (e.g., from the augmentation system 206). The interactive client 104 can present relevant AR effects to the user based on the relevant ranking of the relevant AR effects (e.g., some or all of the AR effects in Table 3), thereby allowing the user to select one of the AR effects to apply to a specific situation.

[0184] In response to the matching of AR effect 1002 with a search query and the user's selection of it from multiple options, interactive client 104 can enable the presentation of image data at mobile device 114 and apply AR effect 1002 to the image data, as described above.

[0185] Therefore, the techniques described in this paper provide useful metadata describing specific AR effects to facilitate uses such as searching for AR effects, generating AR effect recommendations, or ranking AR effects based on relevance.

[0186] Machine Learning Examples

[0187] Figure 11 This is a flowchart depicting a machine learning pipeline 1100 based on some examples. The machine learning pipeline 1100 can be used to generate trained models, for example... Figure 12 The trained machine learning program 1202 is shown in Figure 1200.

[0188] In a broad sense, machine learning can involve automatically learning patterns and relationships in data using computer algorithms, potentially without explicit programming. Machine learning algorithms can be categorized into three main types: supervised learning, unsupervised learning, and reinforcement learning.

[0189] Supervised learning involves training a model using labeled data to predict outputs for new, unseen inputs. Examples of supervised learning algorithms can include linear regression, decision trees, and neural networks.

[0190] Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships in the data. Examples of unsupervised learning algorithms can include clustering, principal component analysis, and generative models (such as autoencoders).

[0191] Reinforcement learning involves training a model to make decisions in dynamic environments by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms can include Q-learning and policy gradient methods.

[0192] Examples of specific machine learning algorithms that can be deployed include logistic regression, a type of supervised learning algorithm used for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example type of machine learning algorithm is Naive Bayes, a supervised learning algorithm used for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random forest is another type of supervised learning algorithm used for classification, regression, and other tasks. Random forests build an ensemble of decision trees and combine their outputs to make predictions. Further examples include neural networks, which consist of layers of interconnected nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another type of machine learning algorithm used for recommendation systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to reveal hidden patterns or relationships in the data. Support vector machines (SVMs) are a type of supervised learning algorithm used for classification, regression, and other tasks. SVMs find hyperplanes that separate different classes in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformer models. The choice of algorithm can depend on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0193] The performance of a machine learning model can be evaluated on a separate test dataset that was not used during training, ensuring that the model can generalize to new, unseen data.

[0194] While this paper discusses several specific examples of machine learning algorithms, at least some of the principles discussed can be applied to other machine learning algorithms. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and Transformers, as well as more traditional machine learning algorithms such as decision trees, random forests, and gradient boosting, can be used in a wide variety of machine learning applications.

[0195] Two typical types of problems in machine learning are classification problems and regression problems. Classification problems (also known as categorization problems) aim to classify an item into one of several category values ​​(e.g., is the object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing values ​​as real numbers).

[0196] Generating a trained machine learning program 1202 may include multiple stages forming part of a machine learning pipeline 1100, said multiple stages including, for example... Figure 11 The following stages are shown:

[0197] Data Collection and Preprocessing 1102: This stage may include acquiring and cleaning data to ensure it is suitable for use in machine learning models. This stage may also include removing duplicates, handling missing values, and converting the data to a suitable format.

[0198] Feature engineering 1104: This stage may include selecting and transforming training data 1206 to create features useful for predicting the target variable. Feature engineering may include (1) receiving features 1208 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying features 1208 in training data 1206 (e.g., unstructured or unlabeled data for unsupervised learning).

[0199] Model selection and training 1106: This stage may include selecting an appropriate machine learning algorithm and training it on preprocessed data. This stage may also involve splitting the data into training and test sets, using cross-validation to evaluate the model, and tuning hyperparameters to improve performance.

[0200] Model Evaluation 1108: This phase may include evaluating the performance of the trained model (e.g., a trained machine learning program 1202) on a separate test dataset. This phase can help determine whether the model is overfitting or underfitting, and whether the model is suitable for deployment.

[0201] Prediction 1110: This stage involves using a trained model (e.g., a trained machine learning program 1202) to generate predictions for new, unseen data.

[0202] Validation, refinement, or retraining 1112: This stage may include updating the model based on feedback generated from the prediction stage (such as new data or user feedback).

[0203] Deployment 1114: This phase may include integrating the trained model (e.g., a trained machine learning program 1202) into a wider system or application (e.g., a web service, mobile application, or Internet of Things (IoT) device). This phase may involve installing the API, building the user interface, and ensuring that the model is scalable and can handle large amounts of data.

[0204] Figure 12 Further details of two example phases are shown: training phase 1204 (e.g., part of model selection and training 1106) and prediction phase 1210 (part of prediction 1110). Prior to training phase 1204, feature engineering 1104 is used to identify features 1208. This can include identifying informative, distinctive, and independent features for efficient operation of the trained machine learning program 1202 in pattern recognition, classification, and regression. In some examples, training data 1206 includes labeled data known for the pre-identified features 1208 and one or more outcomes. Each of the features 1208 can be a variable or attribute, such as various measurable properties of a process, item, system, or phenomenon represented by the dataset (e.g., training data 1206). By way of example only, features 1208 can also have different types such as numerical features, strings, and graphs, and can include one or more of content 1212, concepts 1214, attributes 1216, historical data 1218, and / or user data 1220.

[0205] In training phase 1204, the machine learning program can use training data 1206 to find correlations between features 1208 that influence prediction outcomes or prediction / inference data 1222. Using training data 1206 and the identified features 1208, the trained machine learning program 1202 is trained during training phase 1204, which is part of machine learning program training 1224. Machine learning program training 1224 evaluates the values ​​of features 1208 when they are correlated with training data 1206. The result of the training is the trained machine learning program 1202 (e.g., a trained or learned model).

[0206] Furthermore, the training phase 1204 may involve machine learning, where the training data 1206 is structured (e.g., labeled during preprocessing). The trained machine learning program 1202 may implement a neural network 1226 capable of performing operations such as classification and clustering. In other examples, the training phase 1204 may involve deep learning, where the training data 1206 is unstructured, and the trained machine learning program 1202 implements a deep neural network 1226 capable of performing both feature extraction and classification / clustering operations.

[0207] In some examples, neural network 1226 may be generated during training phase 1204 and implemented within a trained machine learning program 1202. Neural network 1226 includes a hierarchical (e.g., layered) organization of neurons, where each layer consists of multiple neurons or nodes. Neurons in the input layer receive input data, while neurons in the output layer produce the network's final output. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0208] Each neuron in neural network 1226 can operationally compute a function, such as an activation function, which takes as input a weighted sum of the outputs of neurons in the previous layer and a bias term. The output of this function is then passed as input to neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is passed from that neuron (e.g., a sending neuron) to a connecting neuron (e.g., a receiving neuron) in a subsequent layer. The connections between neurons have associated weights that define the influence of the input from the sending neuron to the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the network's performance. Different types of neural networks can use different activation functions and learning algorithms, affecting their performance on different tasks. The hierarchical organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs and generalize to new inputs not seen during training.

[0209] In some examples, neural network 1226 can also be one of several different types of neural networks, such as a single-layer feedforward network, a multilayer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a symmetric connection neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a Transformer network, as examples only. Some machine learning models may include multiple neural networks 1226.

[0210] In addition to the training phase 1204, a validation phase can be performed on a separate dataset called the validation dataset. The validation dataset is used to tune the model's hyperparameters, such as the learning rate and regularization parameters. Tune the hyperparameters to improve the model's performance on the validation dataset.

[0211] Once the model is fully trained and validated, the testing phase allows it to be tested on a new dataset. The test dataset is used to evaluate the model's performance and ensure that it has not overfitted the training data.

[0212] In the prediction phase 1210, the trained machine learning program 1202 uses features 1208 to analyze the query data 1228 to generate inference, results, or predictions, as examples of prediction / inference data 1222. For example, during the prediction phase 1210, the trained machine learning program 1202 generates outputs. The query data 1228 is provided as input to the trained machine learning program 1202, and the trained machine learning program 1202 generates prediction / inference data 1222 as output in response to receiving the query data 1228.

[0213] In some examples, the trained machine learning program 1202 can be a generative artificial intelligence (AI) model. Generative AI is a term that can refer to any type of artificial intelligence that can create new content. For example, generative AI can produce text, images, videos, audio, code, or synthetic data. In some examples, the generated content may resemble the original data, but is not exactly the same.

[0214] Some of the techniques that can be used in generative AI are:

[0215] CNNs: CNNs can be used for image recognition and computer vision tasks. For example, a CNN can be designed to extract features from an image by using filters or kernels that scan the input image and highlight important patterns.

[0216] RNNs: For example, RNNs can be used to process sequential data such as speech, text, and time series data. RNNs employ feedback loops, allowing them to capture temporal dependencies and remember past inputs.

[0217] GAN: A GAN can consist of two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can "fool" the discriminator network, while the discriminator network tries to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.

[0218] Variational Autoencoders (VAEs): VAEs encode input data into a latent space (e.g., a compressed representation) and then decode it back to output data. The latent space can be manipulated to generate new variations of the output data. VAEs can use self-attention mechanisms to process input data, allowing them to handle long text sequences and capture complex dependencies.

[0219] Transformer models: Transformer models use attention mechanisms to learn relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data such as text or speech, as well as non-sequential data such as images or code.

[0220] In the generative AI example, the prediction / inference data 1222 may include predictions, translations, summaries, answers, media content, or a combination thereof.

[0221] The examples described in this article can utilize visual semantic machine learning models. In some examples, the visual semantic machine learning model consists of an image encoder, a text encoder, and a device for fusing information from both encoders. The image and text encoders can use a transformer architecture.

[0222] BLIP-2 model

[0223] As mentioned above, BLIP and BLIP-2 are non-limiting examples of visual semantic machine learning models with multimodal capabilities. BLIP-2 is described below to illustrate certain aspects that can be applied to the implementation of visual semantic machine learning models. However, it should be understood that other types of visual semantic machine learning models can be used to implement the techniques described herein. For example, other multimodal models incorporating LLM can be used to perform such techniques.

[0224] BLIP-2 is a scalable multimodal pre-training method that enables LLMs to ingest and understand images. For example, given an input image and a cue (e.g., a text instruction), a BLIP-2 model can generate a natural language response based on the cue. For instance, given an image of the Great Wall of China and the cue "Tell me some history about this place," the BLIP-2 model can respond by providing a textual description indicating when and why the Great Wall was built. As another example, given an image of a dog and the cue "Describe this animal," the model can respond by stating that the animal is a dog and describing its characteristics.

[0225] BLIP-2 utilizes a frozen visual model in the form of an image encoder and a frozen language model in the form of an LLM. In the context of an LLM, the LLM may not have seen any images during its natural language pre-training, thus creating the so-called "visual-language modality gap." Bridging this gap can be challenging, especially when the LLM is still frozen. BLIP-2 offers methods to bridge this gap, for example, by enabling the LLM to understand visual content through a query transformer (also known as a Q-formulator), which is pre-trained using a two-stage pre-training strategy.

[0226] In some examples, the Q-former is the only trainable part of the BLIP-2 model. The Q-former is connected to both a visual model (e.g., an image encoder) and a language model (e.g., an LLM). The Q-former can include two sub-components:

[0227] An image transformer interacts with a frozen image encoder to extract visual features. A Q-former can extract information from the frozen image encoder using a set of trainable query vectors (e.g., a fixed number of output features, independent of the input image resolution).

[0228] A text transformer that can function as both a text encoder and a text decoder.

[0229] The Q-former can be trained to extract the most informative visual representation of text. In the BLIP-2 strategy, the following three pre-training objectives can be optimized:

[0230] Image-Text Contrast Learning (ITC). This type of learning is used to learn how to align image and text representations to maximize their mutual information.

[0231] Image-based text generation (ITG). This type of learning is used to teach a Q-former how to generate text.

[0232] Image-Text Matching (ITM). This type of learning is used to teach fine-grained alignment between image representations and text representations.

[0233] The first pre-training phase can be referred to as visual and language representation learning. In this phase, the Q-former is connected to a frozen visual model (e.g., an image encoder) and pre-trained with image-text pairs. The Q-former learns to extract image features most relevant to the corresponding text.

[0234] The second pre-training phase can be termed vision-to-language generative learning. In this phase, the output of the Q-former is connected to a frozen language model (e.g., an LLM). The Q-former is trained so that its output features can be interpreted by the LLM to generate the corresponding text. The LLM used in the BLIP-2 model can be either a decoder-based LLM or an encoder-based LLM.

[0235] After pre-training, the Q-former can effectively act as a bridge between the frozen visual model and the frozen language model, thus bridging the visual-language modality gap. During inference, the BLIP-2 model can receive images and cues (e.g., text instructions). The model appends the text instructions to the output of the Q-former as input to the LLM.

[0236] The output of the Q-former can be linearly projected to match the size of the LLM text embedding. This allows the LLM embedding to extract visual information from the Q-former. Therefore, a visual model can extract features from an image, where the Q-former selects relevant information while filtering out information irrelevant to the LLM. Visual semantic machine learning models (such as the BLIP-2 model designed and trained as described above) can be used for various purposes such as VQA, image annotation, or image-text retrieval.

[0237] Data communication architecture

[0238] Figure 13 This is a schematic diagram illustrating the structure of message 1300 according to some examples, which is generated by interactive client 104 for transmission to another interactive client 104 via interactive server 124. The content of a particular message 1300 can be used to populate a file stored in... Figure 1 The message table 304 is located within database 128 and is accessible by interactive server 124. Similarly, the content of message 1300 can be stored in memory as "in-transit" or "in-flight" data for user system 102 or interactive server 124. Message 1300 is shown as including the following example components:

[0239] Message Identifier 1302: A unique identifier that identifies message 1300.

[0240] Message text payload 1304: The text to be generated by the user via the user interface of user system 102 and included in message 1300.

[0241] Message image payload 1306: Image data captured by the camera device component of user system 102 or retrieved from the memory component of user system 102 and included in message 1300. The image data for the sent or received message 1300 can be stored in image table 314.

[0242] Message video payload 1308: Video data captured by the camera device component or retrieved from the memory component of the user system 102 and included in message 1300. The video data for the sent or received message 1300 can be stored in video table 312.

[0243] Message audio payload 1310: Audio data captured by the microphone or retrieved from the memory component of the user system 102 and included in message 1300.

[0244] Message enhancement data 1312: Represents enhancement data (e.g., filters, stickers, or other annotations or enhancements) to be applied to the message image payload 1306, message video payload 1308, or message audio payload 1310 of message 1300. Enhancement data for the sent or received message 1300 can be stored in enhancement table 310.

[0245] Message duration parameter 1314: A parameter value, in seconds, indicating the amount of time that the content of the message (e.g., message image payload 1306, message video payload 1308, message audio payload 1310) will be presented to the user via the interactive client 104 or made accessible to the user.

[0246] Message geolocation parameter 1316: Geolocation data (e.g., latitude and longitude coordinates) associated with the message's content payload. Multiple message geolocation parameter 1316 values ​​may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within the message image payload 1306 or a specific video within the message video payload 1308).

[0247] Message collection identifier 1318: An identifier value that identifies one or more content sets (e.g., “story” identified in set table 316) associated with a specific content item in the message image payload 1306 of message 1300. For example, the identifier value can be used to associate multiple images within the message image payload 1306 with multiple content sets, respectively.

[0248] Message Tag 1320: Each message 1300 can be labeled with multiple tags, each of which indicates the subject of the content included in the message payload. For example, in the case where a specific image depicts an animal (e.g., a lion) is included in the message image payload 1306, a tag value can be included within the message tag 1320 indicating the relevant animal. Tag values ​​can be manually generated based on user input, or can be automatically generated using, for example, image recognition.

[0249] Message sender identifier 1322: An identifier (e.g., message sending system identifier, email address, or device identifier) ​​indicating the user of the user system 102 on which message 1300 is generated and from which message 1300 is sent.

[0250] Message receiver identifier 1324: An identifier (e.g., message sending and receiving system identifier, email address, or device identifier) ​​indicating the user of the user system 102 to which message 1300 is addressed.

[0251] The content (e.g., values) of each component of message 1300 can be pointers to locations in tables where content data values ​​are stored. For example, the image value in message image payload 1306 can be a pointer to a location within image table 314 (or the address of a location within image table 314). Similarly, the value in message video payload 1308 can point to data stored in video table 312, the value stored in message enhancement 1312 can point to data stored in enhancement table 310, the value stored in message collection identifier 1318 can point to data stored in collection table 316, and the values ​​stored in message sender identifier 1322 and message receiver identifier 1324 can point to user records stored in entity table 306.

[0252] Figure 14 A network environment 1400 is shown, according to some examples, in which a head-worn wearable device 1402 (e.g., a head-worn XR or AR device) can be implemented. Figure 14 A high-level functional block diagram of an example head-mounted wearable device 1402, communicatively coupled to a mobile user equipment 1438 and a server system 1432 via a suitable network 1440, is provided. Similar devices can be used... Figure 14 The head-mounted wearable devices 1402 or network of devices shown herein perform one or more of the techniques described herein.

[0253] The head-mounted wearable device 1402 includes at least one of a camera device, such as a visible light camera 1412, an infrared camera, and a transmitter 1414. The head-mounted wearable device 1402 includes other sensors 1416, such as motion sensors or eye-tracking sensors. A user equipment 1438 may be able to connect to the head-mounted wearable device 1402 using both communication links 1434 and 1436. The user equipment 1438 is connected to a server system 1432 via a network 1440. The network 1440 may include any combination of wired and wireless connections.

[0254] The head-mounted wearable device 1402 includes a display arrangement with several components. This arrangement includes two image displays 1404 with optical components. In other examples, the head-mounted wearable device 1402 may then include one image display or more than two image displays.

[0255] The two displays include one associated with the left side of the head-worn device 1402 and one associated with the right side of the head-worn device 1402. The head-worn device 1402 also includes an image display driver 1408, an image processor 1410, a low-power circuitry system 1426, and a high-speed circuitry system 1418. The image display 1404 is used to present images and videos to the user of the head-worn device 1402, including images that can provide a graphical user interface.

[0256] Image display driver 1408 commands and controls the image display of each image display 1404. Image display driver 1408 may deliver image data directly to each image display in image display 1404 for presentation, or may need to convert the image data into a signal or data format suitable for delivery to each image display device. For example, image data may be video data formatted according to a compression format (e.g., H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc.), and still image data may be formatted according to a compression format (e.g., Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tag Image File Format (TIFF), or Exchangeable Image File Format (Exif), etc.).

[0257] The head-wearable device 1402 may include a frame and a stem (or temple) extending from the side of the frame, or another component to facilitate the user wearing the head-wearable device 1402. Figure 14The head-mounted wearable device 1402 also includes a user input device 1406 (e.g., a touch sensor or a press button), which includes an input surface on the head-mounted wearable device 1402. The user input device 1406 is configured to receive input selections from the user to manipulate a graphical user interface of the presented image.

[0258] Figure 14 The components shown for the head-wearable device 1402 are located on one or more circuit boards (e.g., printed circuit boards (PCBs) or flexible PCBs) in the frame or legs. Alternatively or additionally, the depicted components may be located in the blocks, frames, hinges, or nose bridge of the head-wearable device 1402. The left and right sides of the head-wearable device 1402 may each include digital imaging elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0259] The head-mounted wearable device 1402 includes a memory 1422 that stores instructions for performing a subset or all of the functions described herein. The memory 1422 may also include a storage device. For example... Figure 14 As further shown, the high-speed circuit system 1418 includes a high-speed processor 1420, a memory 1422, and a high-speed wireless circuit system 1424. Figure 14 In this embodiment, the image display driver 1408 is coupled to the high-speed circuit system 1418 and operated by the high-speed processor 1420 to drive the left and right image displays of the image display 1404. The high-speed processor 1420 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the head-worn device 1402. The high-speed processor 1420 includes the processing resources required to manage high-speed data transmission over a communication link 1436 to a wireless local area network (WLAN) using the high-speed wireless circuit system 1424. In some examples, the high-speed processor 1420 executes an operating system (e.g., a LINUX operating system or another such operating system for the head-worn device 1402), and the operating system is stored in memory 1422 for execution. Among other duties, the high-speed processor 1420, which executes the software architecture of the head-worn device 1402, manages data transmission with the high-speed wireless circuit system 1424. In some examples, the high-speed wireless circuit 1424 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 1402.11 communication standard (also referred to herein as Wi-Fi). TM In other examples, other high-speed communication standards can be implemented using the high-speed wireless circuit 1424.

[0260] The low-power wireless circuit system 1430 and high-speed wireless circuit system 1424 of the head-mounted wearable device 1402 may include a short-range transceiver (Bluetooth). TM ) and wireless wide area network transceivers, wireless local area network transceivers, or wide area network transceivers (e.g., cellular or Wi-Fi) TM User equipment 1438, including transceivers that communicate via communication links 1434 and 1436, can be implemented using the architectural details of head-mounted wearable device 1402, as can other components of network 1440.

[0261] Memory 1422 includes any storage device capable of storing various data and applications, including camera data generated by the visible light imaging device 1412, sensor 1416, and image processor 1410, and images generated by image display driver 1408 for display on image display 1404, etc. While memory 1422 is shown as integrated with high-speed circuitry 1418, in other examples, memory 1422 may be a separate, independent component of head-mounted wearable device 1402. In some such examples, circuit wiring may provide a connection from image processor 1410 or low-power processor 1428 to memory 1422 via a chip including high-speed processor 1420. In other examples, high-speed processor 1420 may manage addressing of memory 1422 such that low-power processor 1428 will activate high-speed processor 1420 whenever a read or write operation involving memory 1422 is required.

[0262] like Figure 14 As shown, the low-power processor 1428 or high-speed processor 1420 of the head-worn wearable device 1402 may be coupled to a camera device (visible light camera 1412, infrared camera, and transmitter 1414), an image display driver 1408, a user input device 1406 (e.g., a touch sensor or a press button), and a memory 1422. The head-worn wearable device 1402 also includes a sensor 1416, for example, as referenced below. Figure 15 The sensor 1416 may be a motion component 1530, a positioning component 1534, an environmental component 1532, and a biometric component 1528. Specifically, the head-worn device 1402 uses the motion component 1530 and the positioning component 1534 in conjunction with a video feed from one of the visible light camera devices 1412 to determine and track the position and orientation (“posture”) of the head-worn device 1402 relative to a reference frame or another object using techniques such as structure of motion (SfM) or VIO.

[0263] In some examples, and such as Figure 14As shown, the head-mounted wearable device 1402 is connected to a host computer. For example, the head-mounted wearable device 1402 is paired with a user device 1438 via a communication link 1436, or connected to a server system 1432 via a network 1440. The server system 1432 may be one or more computing devices as part of a service or network computing system, for example, it includes a processor, memory, and a network communication interface to communicate with the user device 1438 and the head-mounted wearable device 1402 via the network 1440.

[0264] User equipment 1438 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via network 1440, communication link 1434, or communication link 1436. User equipment 1438 may also store at least a portion of the instructions for implementing the functions described herein.

[0265] The output components of the head-worn wearable device 1402 include visual components, such as displays (e.g., one or more liquid crystal displays (LCDs)), one or more plasma display panels (PDPs), one or more light-emitting diode (LED) displays, one or more projectors, or one or more waveguides. The image display 1404 of the optical components is driven by an image display driver 1408. The output components of the head-worn wearable device 1402 also include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. The input components (e.g., user input device 1406) of the head-worn wearable device 1402, user equipment 1438, and server system 1432 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0266] The head-mounted wearable device 1402 may optionally include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the head-mounted wearable device 1402. For example, peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such components described herein.

[0267] For example, biometric components include those for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Position components include positioning sensor components (e.g., GPS receiver components) for generating position coordinates, and Wi-Fi for generating positioning system coordinates. TM Or Bluetoot TM Transceivers, altitude sensor components (e.g., altimeters or barometers that detect air pressure and determine altitude), orientation sensor components (e.g., magnetometers), etc. The coordinates of such a positioning system can also be received from user equipment 1438 via communication link 1436 through a low-power wireless circuit system 1430 or a high-speed wireless circuit system 1424.

[0268] Machine architecture

[0269] Figure 15This is a schematic representation of machine 1500, within which instructions 1502 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 1500 to perform any or more of the methods discussed herein. For example, instructions 1502 can cause machine 1500 to perform any or more of the methods described herein. Instructions 1502 transform the general, unprogrammed machine 1500 into a specific machine 1500 programmed to perform the described and illustrated functions in the described manner. Machine 1500 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1500 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1500 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), AR devices, XR devices, virtual reality (VR) devices, smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1502 specifying actions to be taken by machine 1500. Furthermore, although only a single machine 1500 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 1502 to perform any or more of the methods discussed herein. For example, machine 1500 may include user system 102 or any of a plurality of server devices forming part of interactive server system 110. In some examples, machine 1500 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0270] Machine 1500 may include a processor 1504, a memory 1506, and an input / output (I / O) unit 1508 that can be configured to communicate with each other via a bus 1510. In the example, processor 1504 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 1512 and 1514 that execute instruction 1502. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 15 Multiple processors 1504 are shown, but machine 1500 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0271] Memory 1506 includes main memory 1516, static memory 1518, and memory cell 1520, all of which are accessible by processor 1504 via bus 1510. Main memory 1506, static memory 1518, and memory cell 1520 store instructions 1502 that implement any one or more of the methods or functions described herein. Instructions 1502 may also reside wholly or partially in main memory 1516, static memory 1518, machine-readable medium 1522 within memory cell 1520, at least one processor of processor 1504 (e.g., within the processor's cache memory), or any suitable combination thereof during execution by machine 1500.

[0272] I / O component 1508 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurement results, etc. The specific I / O component 1508 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It should be recognized that I / O component 1508 may include... Figure 15Many other components are not shown. In various examples, I / O component 1508 may include user output component 1524 and user input component 1526. User output component 1524 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input component 1526 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and force of a touch or touch gesture), audio input components (e.g., microphones), etc.

[0273] In other examples, I / O component 1508 may include biometric component 1528, motion component 1530, environmental component 1532, or position component 1534, as well as a wide range of other components. For example, biometric component 1528 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition).

[0274] Any biometric data collected by the biometric component is captured and stored only with user approval and is deleted upon user request. Furthermore, such biometric data may be used for very limited purposes (e.g., identification and verification). To ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if access to the data occurs). Any use of biometric data may be strictly limited to identification and verification purposes, and such biometric data is not shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0275] The moving part 1530 includes an acceleration sensor part (e.g., an accelerometer), a gravity sensor part, and a rotation sensor part (e.g., a gyroscope).

[0276] The environmental component 1532 includes, for example, one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0277] Regarding the camera device, user system 102 may have a camera device system including, for example, a front-facing camera on the front surface of user system 102 and a rear-facing camera on the rear surface of user system 102. The front-facing camera may be used, for example, to capture still images and videos (e.g., "selfies") of the user of user system 102, which can then be enhanced with the aforementioned enhancement data (e.g., filters). For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera device mode, which are similarly enhanced with enhancement data. In addition to the front-facing and rear-facing cameras, user system 102 may also include a 360° camera for capturing 360° photos and videos.

[0278] Furthermore, the camera system of user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quadruple, or quintuple rear camera configurations on the front and rear sides of user system 102. For example, these multi-camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0279] The position component 1534 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0280] Various technologies can be used to achieve communication. I / O component 1508 also includes communication component 1536, which is operable to couple machine 1500 to network 1538 or device 1540 via a corresponding coupling or connection. For example, communication component 1536 may include a network interface component or another suitable device that interfaces with network 1538. In further examples, communication component 1536 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, Bluetooth component, etc.TM Components (e.g., Bluetooth) TM Low-power, Wi-Fi components, and other communication components for providing communication via other modes. Device 1540 can be any peripheral device from another machine or various peripheral devices (e.g., a peripheral device coupled via USB).

[0281] Furthermore, communication component 1536 can detect identifiers, or includes components operable to detect identifiers. For example, communication component 1536 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, QR codes such as Quick Response (QR) codes, Aztec codes, data matrices, and data symbols). TM The system can utilize optical sensors for multidimensional barcodes and other optical codes, such as MaxiCode, PDF417, UltraCode, and UCC RSS-2D barcodes, or acoustic detection components (e.g., microphones for identifying audio signals of the tags). Additionally, various information can be obtained via communication component 1536, such as location obtained via Internet Protocol (IP) geolocation, location obtained via Wi-Fi® signal triangulation, or location obtained by detecting NFC beacon signals that indicate a specific location.

[0282] Various memories (e.g., main memory 1516, static memory 1518, and the memory of processor 1504) and storage unit 1520 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1502) may, when executed by processor 1504, enable various operations to implement the disclosed examples.

[0283] Instructions 1502 can be sent or received over network 1538 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 1536) and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1502 can be sent or received via a transmission medium coupled to device 1540 (e.g., peer-to-peer coupling).

[0284] Software Architecture

[0285] Figure 16This is a block diagram 1600 illustrating a software architecture 1602 that can be installed on any one or more of the devices described herein. The software architecture 1602 is supported by hardware such as a machine 1604 including a processor 1606, memory 1608, and I / O components 1610. In this example, the software architecture 1602 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 1602 includes layers such as an operating system 1612, libraries 1614, frameworks 1616, and applications 1618. Operationally, application 1618 activates API calls 1620 via the software stack and receives messages 1622 in response to API calls 1620.

[0286] Operating system 1612 manages hardware resources and provides public services. Operating system 1612 includes, for example, a kernel 1624, services 1626, and drivers 1628. Kernel 1624 acts as an abstraction layer between the hardware layer and other software layers. For example, kernel 1624 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1626 can provide other public services to other software layers. Drivers 1628 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1628 may include display drivers, camera drivers, and bluetooth drivers. TM Or BLUETOOTH TM Low-power drives, flash drives, serial communication drives (e.g., USB drives), Wi-Fi drives, audio drives, power management drives, etc.

[0287] Library 1614 provides common low-level infrastructure used by application 1618. Library 1614 may include system library 1630 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1614 may include API library 1632, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing capabilities), and so on. Library 1614 may also include various other libraries 1634 to provide many other APIs to application 1618.

[0288] Framework 1616 provides common high-level infrastructure for use by application 1618. For example, Framework 1616 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 1616 can provide a wide range of other APIs that can be used by application 1618, some of which may be specific to a particular operating system or platform.

[0289] In the example, application 1618 may include home application 1636, contact application 1638, browser application 1640, book reader application 1642, location application 1644, media application 1646, messaging application 1648, game application 1650, and a wide variety of other applications such as third-party application 1652. Application 1618 is a program that performs the functions defined in the program. One or more applications 1618 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1652 (e.g., an entity other than the vendor of a particular platform using Android) TM or iOS TM Applications developed using the SDK can be used in systems such as iOS. TM ANDROID TM Mobile software running on the Windows® Phone mobile operating system or other mobile operating systems. In this example, third-party application 1652 can activate API call 1620 provided by operating system 1612 to facilitate the functionality described herein.

[0290] Example

[0291] In view of the above-mentioned implementation methods, this application discloses the following list of examples, wherein a feature of a single example or more than one feature of an example is combined together, and optionally, is combined with one or more features of one or more other examples, which are also further examples falling within the disclosure of this application.

[0292] Example 1 is a system comprising: at least one processor; and at least one memory unit storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: accessing a first image and a second image, the second image being generated by applying an augmented reality (AR) effect to the first image; providing the first image, the second image, and a cue to a visual semantic machine learning model to obtain an output describing at least one feature of the AR effect; generating a description of the AR effect based on the output of the visual semantic machine learning model; and storing the description in association with an identifier of the AR effect.

[0293] In Example 2, the subject of Example 1 includes, and the operation further includes: applying the AR effect to the first image to obtain the second image.

[0294] In Example 3, the subject of any one of Examples 1 to 2 includes providing the first image and the second image as a stitched image to the visual semantic machine learning model.

[0295] In Example 4, the subject of Example 3 includes, and the operation further includes: generating the stitched image by stitching the first image and the second image together.

[0296] In Example 5, the subject of Example 4 includes, wherein stitching the first image and the second image comprises positioning the first image and the second image relative to each other in a predetermined spatial arrangement to obtain the stitched image.

[0297] In Example 6, the subject of Example 5 includes, wherein the predetermined spatial arrangement includes horizontal splicing.

[0298] In Example 7, the subject of any one of Examples 1 to 6 includes, wherein the first image is a frame of a base video and the second image is a frame of an enhanced video, the enhanced video being generated by rendering the AR effect on the base video.

[0299] In Example 8, the subject of Example 7 includes the operation further comprising: identifying the frame of the enhanced video as corresponding in time to the frame of the base video; extracting the frame of the base video to obtain the first image; and extracting the frame of the enhanced video to obtain the second image.

[0300] In Example 9, the subject of any one of Examples 1 to 8 includes, wherein the prompt is a first prompt, the output is a first output, and the at least one feature includes a first visual feature of the AR effect, and the operation further includes: providing a second prompt to the visual semantic machine learning model to obtain a second output describing a second visual feature of the AR effect, wherein the description of the AR effect is generated based on the first output and the second output.

[0301] In Example 10, the subject of any of Examples 1 to 9 includes, wherein the prompt includes a request to compare the first image and the second image.

[0302] In Example 11, the subject of any one of Examples 1 to 10 includes, wherein the prompt includes a request describing the visual transition from the first image to the second image.

[0303] In Example 12, the subject of any one of Examples 1 to 11 includes, wherein the prompt includes a natural language text prompt, and the description includes a natural language text description.

[0304] In Example 13, the subject of Example 12 includes a prompt, wherein the prompt is a prompt from a predetermined set of prompts for generating the natural language text description of the AR effect.

[0305] In Example 14, the subject of any one of Examples 1 to 13 includes, wherein storing the description includes: in an interactive system, indexing the AR effect based on the description.

[0306] In Example 15, the subject of Example 14 includes the operation further comprising: receiving a search query from a user device of a user of the interactive system; matching the AR effect with the search query based on the index of the AR effect; and, in response to the matching of the AR effect with the search query and a user selection of the AR effect, causing image data to be presented at the user device, the AR effect being applied to the image data.

[0307] In Example 16, the subject of any one of Examples 1 to 15 includes the operation further comprising: determining the category of the AR effect based on the description; and storing the category of the AR effect in association with the identifier of the AR effect.

[0308] In Example 17, the subject of any one of Examples 1 to 16 includes, wherein the AR effect is one of a plurality of AR effects supported by an interactive system, and the visual semantic machine learning model is used to generate a description of each of the plurality of AR effects.

[0309] In Example 18, the subject of Example 17 includes storing the description in association with the identifier of the AR effect, which includes storing the description in a database that maps each of the plurality of AR effects to a corresponding description.

[0310] Example 19 is a method comprising: accessing a first image and a second image, the second image being generated by applying an augmented reality (AR) effect to the first image; providing the first image, the second image, and a cue to a visual semantic machine learning model to obtain an output describing at least one feature of the AR effect; generating a description of the AR effect based on the output of the visual semantic machine learning model; and storing the description in association with an identifier of the AR effect.

[0311] Example 20 is one or more non-transitory computer-readable storage media storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: accessing a first image and a second image, the second image being generated by applying an augmented reality (AR) effect to the first image; providing the first image, the second image, and a cue to a visual semantic machine learning model to obtain an output describing at least one feature of the AR effect; generating a description of the AR effect based on the output of the visual semantic machine learning model; and storing the description in association with an identifier of the AR effect.

[0312] Example 21 is at least one machine-readable medium comprising instructions that, when executed by a processing circuitry system, cause the processing circuitry system to perform an operation for implementing any one of Examples 1 to 20.

[0313] Example 22 is a device that includes means for implementing any one of Examples 1 to 20.

[0314] Example 23 is a system for implementing any one of Examples 1 through 20.

[0315] Example 24 is a method for implementing any one of Examples 1 through 20.

[0316] in conclusion

[0317] As used in this disclosure, the term "machine learning model" (or simply "model") can refer to a single, independent model or a combination of models. The term can also refer to a system, component, or module that includes a machine learning model and one or more supporting or supplementary components that do not necessarily perform machine learning tasks.

[0318] As used in this disclosure, phrases such as "at least one of A, B, or C," "at least one of A, B, or C," and "at least one of A, B, and C" should be interpreted as selecting at least one from the group including "A, B, and C." In this disclosure, unless explicitly stated otherwise in conjunction with specific examples, this phrasing does not imply "at least one of A, at least one of B, and at least one of C." As used in this disclosure, the example "at least one of A, B, or C" will cover any of the following selections: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, and {A, B, C}.

[0319] Unless the context explicitly requires it, throughout the specification and claims, the words “comprising,” “including,” etc., shall be interpreted in an inclusive sense rather than an exclusive or exhaustive sense; that is, meaning “including but not limited to.” As used herein, the terms “connection,” “coupled,” or any variation thereof mean any direct or indirect connection or coupling between two or more elements; the coupling or connection between elements may be physical, logical, or a combination thereof. Furthermore, when used in this application, the words “in this document,” “above,” “below,” and words with similar meanings refer to the entire application and not any particular part of the application. Where the context permits, the use of singular or plural terms may also include the plural or singular, respectively. When referring to a list of two or more items, the word “or” covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list. Similarly, with respect to a list of two or more items, the word “and / or” covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list.

[0320] The various features, steps, operations, and processes described herein can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain methods, processing boxes, or operations may be omitted in some implementations.

[0321] While some examples (such as those depicted in the accompanying figures) include a specific order of operations, that order may be changed without departing from the scope of this disclosure. For example, some operations in the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality described in the examples. In other examples, different components of an example device or system implementing the example methods may perform their functions substantially simultaneously or in a specific order.

[0322] Vocabulary

[0323] For example, "carrier signal" refers to any intangible medium or other intangible medium capable of storing, encoding, or carrying instructions executed by a machine and including digital or analog communication signals. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0324] For example, "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0325] For example, "communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can enable any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0326] For example, a “component” refers to a logical or physical entity having boundaries defined by functional or subroutine calls, branch points, APIs, or other technical definitions that partition or modularize a particular processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components, and part of a program that typically performs a related function. A component can constitute a software component (e.g., code implemented on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuit systems temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine), which is uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be appreciated that a decision can be made, for cost and time considerations, whether to implement a hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider the example of a hardware component being temporarily configured (e.g., programmed), without needing to configure or instantiate each hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor that is configured as a dedicated processor via software, this general-purpose processor can be configured at different times as its respective dedicated processor (e.g., including different hardware components). The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as a “Software as a Service” (SaaS) operation. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some examples, the processor or processor-implemented component may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component may be distributed across multiple geographic locations.

[0327] For example, "computer-readable storage medium" refers to both machine-readable storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0328] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0329] For example, a "non-transitory computer-readable storage medium" refers to a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0330] For example, "signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions executable by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.

[0331] "User equipment" means, for example, a device that is accessed, controlled, or owned by a user and that the user interacts with to perform actions or interactions (including interactions with other users or computer systems).

Claims

1. A system comprising: At least one processor; as well as At least one memory unit stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: Access a first image and a second image, the second image being generated by applying augmented reality (AR) effects to the first image; The first image, the second image, and a cue are provided to a visual semantic machine learning model to obtain an output describing at least one feature of the AR effect; Based on the output of the visual semantic machine learning model, a description of the AR effect is generated; and The description is stored in association with the identifier of the AR effect.

2. The system according to claim 1, wherein the operation further includes: The AR effect is applied to the first image to obtain the second image.

3. The system according to claim 1, wherein, The first image and the second image are provided as a stitched image to the visual semantic machine learning model.

4. The system according to claim 3, wherein the operation further includes: The stitched image is generated by stitching together the first image and the second image.

5. The system according to claim 4, wherein, Stitching the first image and the second image involves positioning the first image and the second image relative to each other in a predetermined spatial arrangement to obtain the stitched image.

6. The system according to claim 5, wherein, The predetermined spatial arrangement includes horizontal splicing.

7. The system according to claim 1, wherein, The first image is a frame of a base video, and the second image is a frame of an enhanced video, which is generated by rendering the AR effect on the base video.

8. The system according to claim 7, wherein the operation further comprises: The frames of the enhanced video are identified as corresponding to the frames of the base video in time; Extract frames from the base video to obtain the first image; as well as Frames of the enhanced video are extracted to obtain the second image.

9. The system according to claim 1, wherein, The prompt is a first prompt, the output is a first output, and the at least one feature includes a first visual feature of the AR effect; the operation further includes: A second cue is provided to the visual semantic machine learning model to obtain a second output describing a second visual feature of the AR effect, wherein the description of the AR effect is generated based on the first output and the second output.

10. The system according to claim 1, wherein, The prompt includes a request to compare the first image and the second image.

11. The system according to claim 1, wherein, The prompt includes a request describing the visual transition from the first image to the second image.

12. The system according to claim 1, wherein, The prompt includes a natural language text prompt, and the description includes a natural language text description.

13. The system according to claim 12, wherein, The prompt is one of a predetermined set of prompts used to generate the AR effect from the natural language text description.

14. The system according to claim 1, wherein, The storage of the description includes: In the interactive system, the AR effect is indexed based on the description.

15. The system of claim 14, further comprising: Receive search queries from the user's user device in the interactive system; Based on the index of the AR effect, the AR effect is matched with the search query; as well as In response to the matching of the AR effect with the search query and the user's selection of the AR effect, image data is presented on the user device, and the AR effect is applied to the image data.

16. The system according to claim 1, wherein the operation further comprises: The category of the AR effect is determined based on the description; as well as The category of the AR effect is stored in association with the identifier of the AR effect.

17. The system according to claim 1, wherein, The AR effect is one of a plurality of AR effects supported by the interactive system, and the visual semantic machine learning model is used to generate a description of each of the plurality of AR effects.

18. The system according to claim 17, wherein, Storing the description in association with the identifier of the AR effect includes storing the description in a database that maps each of the plurality of AR effects to a corresponding description.

19. A method comprising: Access a first image and a second image, the second image being generated by applying augmented reality (AR) effects to the first image; The first image, the second image, and a cue are provided to a visual semantic machine learning model to obtain an output describing at least one feature of the AR effect; Based on the output of the visual semantic machine learning model, a description of the AR effect is generated; and The description is stored in association with the identifier of the AR effect.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform an operation, the operation comprising: Access a first image and a second image, the second image being generated by applying augmented reality (AR) effects to the first image; The first image, the second image, and a cue are provided to a visual semantic machine learning model to obtain an output describing at least one feature of the AR effect; Based on the output of the visual semantic machine learning model, a description of the AR effect is generated; and The description is stored in association with the identifier of the AR effect.

Images