Model fine-tuning for automated augmented reality

Abstract

A second input image is generated by applying a target augmented reality (AR) effect to a first input image. The first input image and the second input image are provided to a first visual semantic machine learning model to obtain an output that describes at least one characteristic of the target AR effect. The first visual semantic machine learning model is fine-tuned by a second visual semantic machine learning model by using training samples. Each training sample includes a first training image, a second training image, and a training description of a given AR effect. The second training image is generated by applying the given AR effect to the first training image. The description of the target AR effect is selected based on the output of the visual semantic machine learning model. The description of the target AR effect is stored in association with an identifier of the target AR effect.

Description

Declaration of priority

[0001] This application claims the benefit of priority to U.S. Patent Application Serial No. 18 / 502,868, filed November 6, 2023, which is incorporated herein by reference in its entirety. Technical Field

[0002] The topics disclosed herein generally relate to augmented reality (AR) technology. More specifically, but not exclusively, the topics disclosed herein relate to the generation of descriptions of AR effects for use in interactive systems. Background Technology

[0003] Some interactive applications enable users to apply AR effects (also known as augmentations) to content items, such as images or videos captured by the user. AR effects can be stored in the interactive system along with metadata such as keywords. For example, the interactive system can link AR effects to corresponding keywords, making it easier for users of the interactive application to search for and apply the desired AR effects. Attached Figure Description

[0004] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals can describe similar parts in different views. To facilitate identification of any discussion of a particular element or action, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced. Some non-limiting examples are shown in the accompanying drawings:

[0005] Figure 1 It is a diagrammatic representation of a networked environment in which the content of this disclosure can be deployed, based on some examples.

[0006] Figure 2 It is a graphical representation of an interactive system with both client-side and server-side functionalities, based on some examples.

[0007] Figure 3 It is a graphical representation based on examples such as data structures maintained in a database.

[0008] Figure 4 It is a graphical representation of components of an AR effect description system and an artificial intelligence and machine learning system, based on some examples.

[0009] Figure 5 This is a flowchart illustrating a method for fine-tuning a visual semantic machine learning model, based on some examples.

[0010] Figure 6 It is a graphical representation of the components of a visual semantic machine learning model based on some examples.

[0011] Figure 7The efficient parameter fine-tuning techniques based on some examples are illustrated in a graphical manner.

[0012] Figure 8 This is a flowchart illustrating a method for generating AR effects based on some examples.

[0013] Figure 9 The first image is shown based on some examples.

[0014] Figure 10 This demonstrates, based on some examples, how to apply AR effects. Figure 9 The second image is generated from the first image.

[0015] Figure 11 This illustrates, based on some examples, how to arrange spaces according to a predetermined plan. Figure 9 The first image and Figure 10 The stitched image is generated by stitching together the second image.

[0016] Figure 12 The first image is shown based on some examples.

[0017] Figure 13 This demonstrates, based on some examples, how to apply AR effects. Figure 12 The second image is generated from the first image.

[0018] Figure 14 The machine learning pipeline is illustrated using diagrams based on some examples.

[0019] Figure 15 The training and use of machine learning programs are illustrated using diagrams based on some examples.

[0020] Figure 16 It is a graphical representation based on some example messages.

[0021] Figure 17 The diagram illustrates a network environment in which a head-mounted wearable device can be implemented, based on some examples.

[0022] Figure 18 It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0023] Figure 19 It is a block diagram showing the software architecture in which examples can be implemented. Detailed Implementation

[0024] As used in this disclosure, the term "AR effect" refers to an effect, modification, removal, addition, or combination thereof that alters an image or image sequence (e.g., a video clip including a sequence of image frames) when it is captured and rendered without any such effect, modification, removal, or addition. AR effects may also be referred to as "enhancements." Examples of AR effects may include two-dimensional or three-dimensional effects, filters, lenses, media overlays (e.g., text, color, or image overlays), AR experiences, extended reality (XR) experiences, or combinations thereof.

[0025] AR effects can render special, interesting, entertaining, or useful enhancements on content items such as images or videos. For example, a user of an interactive application can select the AR effects to be applied to video content captured using the user's device's camera. The AR effects can then be applied in real time (e.g., before or during content capture, to objects presented in the camera feed interface) or after content capture (e.g., to video files retrieved from storage locations associated with the user's device).

[0026] For example, AR effects can overlay rendering onto a person's face, render virtual objects in specific locations relative to real-world objects, or change the colors of an image. In some cases, AR effects can include changes to the format or style of content items, such as applying a "green screen" effect to video captured by a user. In others, AR effects can produce both aesthetic and formatting changes.

[0027] As mentioned, AR effects can be stored in interactive systems along with metadata such as keywords or "tags." In some cases, the metadata of a specific AR effect can provide a useful description of the AR effect, facilitating the search for AR effects, the generation of AR effect recommendations, and the ranking of AR effects. However, interactive systems can support a large number of AR effects (e.g., thousands or even millions of different AR effects), making the manual review and labeling of AR effects a challenging task. Therefore, it may be desirable to automate the generation of AR effect descriptions, rather than, for example, requesting human labelers to perform this task.

[0028] It is possible to automatically generate keywords or labels by processing augmented images using object recognition. For example, machine learning models can be trained to detect objects in an image and output predicted labels or categories based on the detected objects. However, this technique may not produce the desired output. For instance, an object recognition machine learning model may correctly detect certain objects in an image that have been augmented by AR effects, but it may fail to distinguish between objects specifically associated with the AR effect and objects that form part of the original or base image. For example, in the case where the AR effect is a "dog face filter" added to a human face, the machine learning model may struggle to separate the dog features from other features in the image or video that are unrelated to the AR effect. Therefore, automated descriptions of AR effects produced in this way may have limited usefulness or relevance.

[0029] The examples described in this paper address or mitigate the technical problems mentioned above by using techniques involving visual semantic machine learning models. In some examples, visual semantic machine learning models are used to provide more accurate, relevant, or comprehensive descriptions of AR effects supported by interactive systems. The techniques described in this paper can allow the automatic generation of AR effect descriptions that decouple the rendered AR effect from the “base” or “background” content, making the descriptions more useful in downstream tasks such as AR effect indexing, searching, ranking, or categorizing. In some examples, AR effect descriptions can serve as descriptive text for AR effects within interactive systems or as labels for AR effects.

[0030] Therefore, visual semantic machine learning models can be trained to mimic human labelers. In some examples, fine-tuning is used to obtain visual semantic machine learning models that can better separate or distinguish the target AR effect from the "base" or "background" content. In some examples, through fine-tuning, visual semantic machine learning models can automatically provide AR effect descriptions based solely on the input image. Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) or prefix tuning can be used for this purpose.

[0031] As used herein, the term "fine-tuning" generally refers to the process of adapting a pre-trained machine learning model. For example, a machine learning model can be adapted to improve its performance on a specific task or to make it more suitable for a specific operation. Fine-tuning techniques can include one or more of the following: updating or changing the internal parameters of a pre-trained model through additional training, injecting new trainable weights or layers into the model architecture and training those weights or layers, modifying the model topology by changing layers or connections, changing aspects of the training process (e.g., loss function or optimization method), or any other adaptation that, for example, produces better model performance for a specific task compared to the pre-trained model.

[0032] One approach may include generating a second input image by applying a target AR effect to a first input image. Both the first and second input images can be accessed and fed to a first visual semantic machine learning model to obtain an output describing at least one feature of the target AR effect. In some examples, the first visual semantic machine learning model is obtained by fine-tuning a second visual semantic machine learning model on training data including multiple training samples.

[0033] Each training sample may include a first training image, a second training image, and a training description of a given AR effect. The second training image is generated by applying the given AR effect to the first training image.

[0034] Based on the output of the first-vision semantic machine learning model, a description of the target AR effect is selected or generated. The description of the target AR effect is stored in association with the identifier of the target AR effect.

[0035] Fine-tuning can be used to optimize a general visual semantic machine learning model pre-trained to generate descriptive text for images, focusing on AR effects present in the images. In some examples, parameter-efficient fine-tuning, such as LoRA, can be utilized. Parameter-efficient fine-tuning techniques are used to reduce the computational cost of fine-tuning.

[0036] In LoRA, the model's weight matrix can be decomposed into a low-rank matrix. This reduces the number of parameters that need to be trained. LoRA processing can include using training samples to update the transformer of the second visual semantic machine learning model to obtain the first visual semantic machine learning model.

[0037] In some examples, the transformer lies between the large language model (LLM) of the first visual semantic machine learning model and the image encoder. Such a transformer can be called a querying transformer (or "Q-former"). Updating the querying transformer can include adjusting its parameters (e.g., attention weights) while keeping the image encoder and LLM fixed.

[0038] The first visual semantic machine learning model can generate outputs that describe at least one visual feature of the target AR effect. For example, the first visual semantic machine learning model is fine-tuned to describe visual features based on visual transitions from a first input image to a second input image caused by the target AR effect. In some examples, the first visual semantic machine learning model uses the first and second input images to perform a captioning operation. Given that the captioning operation is focused on the AR effect, it can be considered AR effect-specific.

[0039] In some examples, a first input image and a second input image are provided to a first visual semantic machine learning model as a stitched image. The stitched image can be generated by stitching the first input image and the second input image together. Stitching the first input image and the second input image may include positioning the first input image and the second input image relative to each other in a predetermined spatial arrangement (e.g., horizontal stitching) to obtain the stitched image.

[0040] In some examples, two image frames are stitched together to provide or simulate a "before and after," which allows visual semantic machine learning models to better analyze the effects of the augmentation. In other examples, more than two image frames may be used (e.g., by presenting a series of frames to illustrate the results of the AR effect).

[0041] In the context of images, the term "stitching" as used herein refers to any image formed by combining or integrating multiple (e.g., two or more) images or portions of multiple images. In some examples, the method involves generating a stitched image by stitching together a first image and a second image.

[0042] During fine-tuning, the first and second training images can be provided as stitched training images to the second visual semantic machine learning model. Similarly, in some examples, the stitched training images can be generated by positioning the first and second training images relative to each other in a predetermined spatial arrangement. Therefore, a "before and after" effect can also be simulated during fine-tuning, thereby fine-tuning the visual semantic machine learning model to focus on the enhancement effect.

[0043] In some examples, the images are video frames. For instance, the first input image is a frame from a base video, and the second input image is a frame from an augmented video, where the augmented video is generated by rendering a target AR effect on the base video. Similarly, the first training image can be a frame from a base video, and the second training image can be a frame from an augmented video, where the augmented video is generated by rendering a given AR effect associated with the training samples on the base video.

[0044] To obtain image pairs (e.g., a first image and a second image) for training or inference, frames of the augmented video that temporally correspond to frames of the base video can be identified. These frames can then be extracted to obtain the associated first and second images.

[0045] In some examples, the frames of the base video and the frames of the enhanced video do not correspond exactly in time. For example, the two videos may have unequal frame durations. Therefore, the term "corresponding in time" should be interpreted to include relatively small or slight differences. In some examples where the frames of the two videos have unequal durations, the frame duration of one video (with a longer duration) may be truncated to correspond to the frame duration of the other video (with a shorter duration).

[0046] As mentioned above, in some examples, a description of the target AR effect is stored in association with an identifier for the target AR effect. The description can be a natural language text description. In some examples, the output of the first visual semantic machine learning model is used directly as the description. In other examples, the output is modified to obtain the description, as described further below.

[0047] Target AR effects can be indexed in an interactive system based on descriptions. One approach may include receiving a search query from a user's device within the interactive system. The search query may be matched against a target AR effect (e.g., based on a match between the search query and a description of the target AR effect indexed in the interactive system). In response to a match between the target AR effect and the search query, and / or a user selection of the target AR effect, the interactive system may cause image data with the target AR effect applied to it to be presented at the user's device.

[0048] This method may include determining the category of a target AR effect based on a description generated for the AR effect. The category may be stored in association with an identifier for the target AR effect.

[0049] Visual semantic machine learning models can be used to generate descriptions of multiple different AR effects. Each description can be stored in a database that matches each AR effect with its corresponding description.

[0050] The examples described in this paper allow for the automatic and large-scale generation of natural language-based descriptions (e.g., searchable descriptive text or tags) of AR effects through image captioning / description capabilities using machine learning models. These descriptions can, for example, be used to automatically index a large number (e.g., thousands) of AR effects within an interactive system. Improving the quality of the descriptions (e.g., by providing descriptions specifically focused on the enhanced visual effects based on "before and after" analysis) can improve the relevance or accuracy of search results.

[0051] The examples described in this article can provide many technical advantages, such as facilitating the retrieval of AR effects from storage locations, improving the quality of search results in interactive applications, allowing for easier generation of insights related to similarities or differences between AR effects, and improving the rating, ranking, recommendation, or classification of AR effect enhancements.

[0052] The examples described in this paper also leverage model fine-tuning to obtain a visual semantic machine learning model with improved ability to recognize or focus on AR effects in images. While manual captioning of AR effects can require significant human effort and result in inconsistent quality among different human labelers, directly applying a visual semantic model (e.g., a fine-tuned model) can produce better and more consistent captioning.

[0053] Furthermore, efficient parameter fine-tuning can achieve efficient use of computational resources. For example, LoRA allows for efficient model specialization while fine-tuning only a small subset of the original model weights, thereby avoiding interference with general knowledge in the pre-trained model and reducing computational resource requirements.

[0054] In some examples, the techniques described herein improve the functionality of content moderation systems by providing more accurate, detailed, or comprehensive descriptions of AR effects in an automated manner. For instance, by applying the automated AR effect description generation methods described herein, computer systems can be improved to accurately identify AR effects (e.g., to propose new AR effects for use in interactive systems) that include prohibited or offensive content.

[0055] When taken into account the effects of this disclosure, one or more of the methods described herein can eliminate the need for certain efforts or resources originally involved in generating or indexing AR effect descriptions or retrieving related AR effects. For example, by providing more accurate or comprehensive AR effect search results or recommendations, the computational resources used by one or more machines, databases, or networks can be utilized more efficiently or even reduced, thereby reducing the need for duplicate searches or further filtering of user searches. Examples of such computational resources may include processor cycles, network traffic, memory usage, graphics processing unit (GPU) resources, data storage capacity, power consumption, and cooling capacity.

[0056] Networked computing environment

[0057] Figure 1This is a block diagram illustrating an example interactive system 100 for facilitating interactions over a network, such as exchanging text messages, making text, audio, and video calls, or playing games. The interactive system 100 includes multiple user systems 102, each of which hosts multiple applications including an interactive client 104 (as an example of an interactive application) and other applications 106. Each interactive client 104 is communicatively coupled to other instances of the interactive client 104 (e.g., hosted on corresponding other user systems 102), an interactive server system 110, and a third-party server 112 via one or more communication networks including a network 108 (e.g., the Internet). The interactive client 104 can also communicate with the locally hosted applications 106 using an application programming interface (API).

[0058] Each user system 102 may include multiple user devices, such as mobile devices 114, head-mounted wearable devices 116, and computer client devices 118, that are communicatively connected to exchange data and messages.

[0059] Interactive client 104 interacts with other interactive clients 104 and with interactive server system 110 via network 108. The data exchanged between interactive clients 104 (e.g., interaction 120) and between interactive client 104 and interactive server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0060] Interactive server system 110 provides server-side functionality to interactive client 104 via network 108. While some functions of interactive system 100 are described herein as being performed by interactive client 104 or interactive server system 110, the location of certain functions within interactive client 104 or interactive server system 110 may be a design choice. For example, it may be technically preferred that certain technologies and functions are initially deployed within interactive server system 110, but later migrated to interactive client 104 where user system 102 has sufficient processing power.

[0061] The interactive server system 110 supports various services and operations provided to the interactive client 104. Such operations include sending data to and receiving data from the interactive client 104, and processing data generated by the interactive client 104. This data may include message content, client device information, geolocation information, content enhancements (e.g., filters and overlays), message content persistence conditions, entity relationship information, and live event information. Data exchange within the interactive system 100 is activated and controlled via functions available through the user interface of the interactive client 104.

[0062] Specifically, turning to interactive server system 110, API server 122 is coupled to interactive server 124 and provides it with a programming interface, making the functionality of interactive server 124 accessible to interactive client 104, other applications 106, and third-party server 112. Interactive server 124 is communicatively coupled to database server 126, thereby facilitating access to database 128, which stores data associated with the interactions processed by interactive server 124. Similarly, web server 130 is coupled to interactive server 124 and provides a web-based interface to interactive server 124. To this end, web server 130 handles incoming network requests via Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0063] API server 122 receives and sends interactive data (e.g., command and message payloads) between interactive server 124 and user system 102 (as well as interactive client 104 and other applications 106) and third-party server 112. Specifically, API server 122 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by interactive client 104 and other applications 106 to activate the functionality of interactive server 124. API server 122 exposes various functions supported by interactive server 124, including account registration; login functionality; sending interactive data from one interactive client 104 to another interactive client 104 via interactive server 124; transferring media files (e.g., images or videos) from interactive client 104 to interactive server 124; setting media data sets (e.g., stories); retrieving the friend list of users in user system 102; retrieving messages and content; retrieving or applying AR effects; adding and deleting entities (e.g., friends) against an entity graph (e.g., entity graph 308); locating friends in the entity graph; and opening (e.g., application events associated with interactive client 104). Interactive server 124 hosts multiple systems and subsystems, as shown below. Figure 2 Describe it.

[0064] Related Applications

[0065] Returning to interactive client 104, the features and functionality of external resources (e.g., linked application 106 or applet) can be made available to the user via the interface of interactive client 104. In this context, "external" refers to the fact that application 106 or applet is outside of interactive client 104. External resources are typically provided by third parties, but may also be provided by the creator or provider of interactive client 104. Interactive client 104 receives user selections regarding options to launch or access the features of such external resources. External resources can be application 106 installed on user system 102 (e.g., a "local application"), or a smaller version (e.g., a "applet") of an application hosted on user system 102 or remotely on user system 102 (e.g., on a third-party server 112). A smaller version of an application includes a subset of the features and functionality of the application (e.g., a full-scale native version of the application) and is implemented using markup language documentation. In some examples, a smaller version of an application (e.g., a "applet") is a web-based markup language version of the application and is embedded in interactive client 104. In addition to using markup language documentation (e.g., ... In addition to files, mini-programs can also include scripting languages ​​(e.g., ... (file or .json file) and stylesheets (e.g.) document).

[0066] In response to a user selection of an option to launch or access an external resource, interactive client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, application 106, locally installed on user system 102, can be launched independently of and separately from interactive client 104, for example, by selecting the icon corresponding to application 106 on the home screen of user system 102. A smaller version of such an application can be launched or accessed via interactive client 104, and in some examples, no part of the smaller application can be accessed outside of interactive client 104, or only a limited portion of the smaller application can be accessed outside of interactive client 104. The smaller application can be launched by interactive client 104 by receiving, for example, markup language documents associated with the smaller application from third-party server 112 and processing such documents.

[0067] In response to determining that the external resource is a locally installed application 106, the interactive client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 104 communicates with a third-party server 112 (e.g.) to obtain a markup language document corresponding to the selected external resource. The interactive client 104 then processes the obtained markup language document to render the web-based external resource within the user interface of the interactive client 104.

[0068] Interactive client 104 can notify users of user system 102 or other users (e.g., "friends") associated with such users of one or more external resources. For example, interactive client 104 can provide participants in a conversation (e.g., a chat session) within interactive client 104 with notifications related to the current or recent use of external resources by one or more members of a group of users. One or more users can be invited to join an active external resource or to activate (in a group of friends) a recently used but currently inactive external resource. External resources can provide participants in the conversation, each using the corresponding interactive client 104, with the ability to share items, statuses, states, or locations within the external resource with one or more members of a group of users during a chat session. Shared items can be interactive chat cards, which chat members can use to interact, for example, activate the corresponding external resource, view specific information within the external resource, or take chat members to a specific location or state within the external resource. Within a given external resource, response messages can be sent to users on interactive client 104. External resources can selectively include different media items in the response based on the current context of the external resource.

[0069] Interactive client 104 can present a list of available external resources (e.g., application 106 or mini-program) to the user to launch or access a given external resource. This list can be presented in a context-sensitive menu. For example, icons representing different applications within application 106 (or mini-program) can vary depending on how the user launches the menu (e.g., from a conversational interface or a non-conversational interface).

[0070] System Architecture

[0071] Figure 2This is a block diagram illustrating further details of an interactive system 100 according to some examples. Specifically, the interactive system 100 is shown as including an interactive client 104 and an interactive server 124. The interactive system 100 includes multiple subsystems supported on the client side by the interactive client 104 and on the server side by the interactive server 124. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) may have components that enable the microservice subsystem to operate independently and communicate with other services. Example components of a microservice subsystem may include:

[0072] Functional logic: Functional logic implements the functions of the microservice subsystem and represents the specific capabilities or functions provided by the microservice.

[0073] API Interfaces: Microservices can use lightweight protocols such as Representational State Transfer (REST) ​​or messaging to communicate with each other through well-defined APIs or interfaces. API interfaces define the inputs and outputs of a microservice subsystem and how it interacts with other microservice subsystems within the interactive system 100.

[0074] Data storage: The microservice subsystem can be responsible for its own data storage, which can be in the form of a database, cache, or other storage mechanisms (e.g., using database server 126 and database 128). This allows the microservice subsystem to operate independently of other microservices in the interactive system 100.

[0075] Service discovery: Microservice subsystems can find other microservice subsystems of the interactive system 100 and communicate with them. The service discovery mechanism enables microservice subsystems to locate and communicate with other microservice subsystems in a scalable and efficient manner.

[0076] Monitoring and logging: Microservice subsystems may need to be monitored and logged to ensure availability and performance. Monitoring and logging mechanisms enable the tracking of the health and performance of microservice subsystems.

[0077] In some examples, the interactive system 100 may employ a monolithic architecture, a service-oriented architecture (SOA), a functionality-as-a-service (FaaS) architecture, or a modular architecture. Example subsystems are... Figure 2 It is shown in the figure and discussed below.

[0078] Image processing system 202 provides various functions that enable users to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. Camera device system 204 includes (e.g., in a camera device application) control software that (e.g., directly or via operating system controls) interacts with and controls the hardware camera device hardware of user system 102 to modify and enhance real-time images captured and displayed via interactive client 104.

[0079] Enhancement system 206 provides functionality related to the generation and distribution of enhancements (e.g., filters or media overlays)—also known as AR effects)—for images or videos captured in real time by the camera device of user system 102 or retrieved from the memory of user system 102. For example, enhancement system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) to interactive client 104 to enhance real-time images received via camera device system 204 or stored images retrieved from the memory of user system 102. These enhancements are selected by enhancement system 206 and presented to the user of interactive client 104 based on several inputs and data, such as:

[0080] The geographic location of user system 102; and

[0081] User entity relationship information of users in user system 102.

[0082] Enhanced or AR effects can include audio and visual content and visual effects. Examples of audio and visual content include images, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at user system 102 for transmission in messages, or to video content such as video content streams or feeds sent from interactive client 104. Therefore, image processing system 202 can interact with and support various subsystems of communication system 208, such as messaging system 210 and video communication system 212.

[0083] Media overlays may include text or image data that can be superimposed on photographs taken by user system 102 or video streams produced by user system 102. In some examples, media overlays may be location overlays (e.g., Venice Beach), names of live events, or names of businesses (e.g., Beach Cafe). In other examples, image processing system 202 uses the geolocation of user system 102 to identify media overlays that include the names of businesses located at the geolocation of user system 102. Media overlays may include additional identifiers associated with the businesses. Media overlays (or links / paths to them) may be stored in database 128 and accessed through database server 126.

[0084] Image processing system 202 provides a user-based publishing platform that allows users to select a geographic location on a map and upload content associated with that location. Users can also specify which media overlays should be provided to other users. Image processing system 202 generates a media overlay that includes the uploaded content and associates it with the selected geographic location.

[0085] The enhancement creation system 214 supports an AR developer platform and includes applications for content creators (e.g., artists and developers) to create and publish enhancements (e.g., AR experiences) to interactive clients 104. The enhancement creation system 214 provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking technologies, and templates. In some examples, the enhancement creation system 214 provides a merchant-based publishing platform that allows merchants to select specific enhancements associated with geolocation via a bidding process. For example, the enhancement creation system 214 associates the media overlay of the highest-bidding merchant with a corresponding geolocation for a predefined amount of time.

[0086] Communication system 208 is responsible for enabling and processing various forms of communication and interaction within interactive system 100, and includes messaging system 210, audio communication system 216, and video communication system 212. In some cases, messaging system 210 is responsible for enabling temporary or time-limited access to content by interactive client 104. Messaging system 210 includes (e.g., in a short-lived timer system) multiple timers that selectively enable access (e.g., for presentation and display) of messages and associated content via interactive client 104 based on duration and display parameters associated with a message or set of messages (e.g., a story). Audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 104.

[0087] User management system 218 is operationally responsible for managing user data and profiles, and maintaining information about users of interactive system 100 and the relationships between users (e.g., stored in...). Figure 3 The entity information shown in Entity Table 306, Entity Diagram 308 and Profile Data 302.

[0088] The collection management system 220 is operationally responsible for managing collections or sets of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period (e.g., the duration of the event to which the content relates). For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 220 can also be responsible for publishing icons that notify the user interface of the interactive client 104 of the availability of specific collections. The collection management system 220 includes curation functions that enable collection managers to manage and curate specific content collections. For example, a curation interface enables event organizers to curate collections of content related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 220 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, users may be compensated for including user-generated content in a collection. In such cases, the collection management system 220 operates to automatically pay such users for using their content.

[0089] Map system 222 provides various geolocation functions and supports the presentation of map-based media content and messages by interactive client 104. For example, map system 222 enables the display (e.g., stored in profile data 302) of user icons or avatars on the map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content generated by these friends (e.g., a collection of messages including photos and videos). For example, on the map interface of interactive client 104, messages posted by a user from a specific geolocation to interactive system 100 can be displayed to the specific user's "friends" within the context of that specific location on the map. Users can also share their location and status information with other users of interactive system 100 via interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of the map interface of interactive client 104.

[0090] Game system 224 provides various game functions within the context of interactive client 104. Interactive client 104 provides a game interface that offers a list of available games that can be initiated by a user within the context of interactive client 104 and played with other users of interactive system 100. Interactive system 100 also enables specific users to invite other users to participate in specific games by sending invitations from interactive client 104. Interactive client 104 also supports sending and receiving voice, video, and text messages (e.g., chat) within the context of playing the game, provides leaderboards for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0091] External resource system 226 provides interactive client 104 with an interface to communicate with remote servers (e.g., third-party server 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts applications or smaller versions of applications (e.g., game applications, utility applications, payment applications, or ride-sharing applications) based on markup languages ​​(e.g., HTML5). Interactive client 104 can launch web-based resources (e.g., applications) by accessing HTML5 files from the third-party server 112 associated with the web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript using a software development kit (SDK) provided by interactive server 124. The SDK includes APIs with functionality that can be called or activated by the web-based application. Interactive server 124 hosts a JavaScript library that provides access to a given external resource for specific user data of interactive client 104. HTML5 is an example of a technology used for programming games, but applications and resources programmed using other technologies can be used.

[0092] To integrate the SDK's functionality into the web-based resource, the third-party server 112 downloads the SDK from the interactive server 124, or the third-party server 112 otherwise receives the SDK. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate the features of the interactive client 104 into the web-based resource.

[0093] The SDK stored on the interactive server system 110 effectively bridges the gap between external resources (e.g., application 106 or applet) and the interactive client 104. This provides users with a seamless experience communicating with other users on the interactive client 104 while preserving the look and feel of the interactive client 104. To bridge communication between the external resources and the interactive client 104, the SDK facilitates communication between a third-party server 112 and the interactive client 104. A bridging script running on the user system 102 establishes two unidirectional communication channels between the external resources and the interactive client 104. Messages are sent asynchronously between the external resources and the interactive client 104 via these communication channels. Each SDK function activation is sent as a message and callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with that callback identifier.

[0094] By using the SDK, not all information from the interactive client 104 is shared with the third-party server 112. The SDK limits which information is shared based on the needs of the external resources. Each third-party server 112 provides the interactive server 124 with an HTML5 file corresponding to the web-based external resource. The interactive server 124 can add a visual representation (e.g., box design or other graphics) of the web-based external resource to the interactive client 104. Once the user selects the visual representation or instructs the interactive client 104 to access the features of the web-based external resource through the graphical user interface of the interactive client 104, the interactive client 104 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0095] Interactive client 104 presents a graphical user interface (GUI) for an external resource (e.g., a login page or title screen). During, before, or after presenting the login page or title screen, interactive client 104 determines whether the initiated external resource has previously been authorized to access user data of interactive client 104. In response to determining that the initiated external resource has previously been authorized to access user data of interactive client 104, interactive client 104 presents another GUI of the external resource, including its functionality and characteristics. In response to determining that the initiated external resource has not previously been authorized to access user data of interactive client 104, after displaying the login page or title screen of the external resource for a threshold time period (e.g., 3 seconds), interactive client 104 slides up a menu (e.g., animates the menu to appear from the bottom of the screen to the middle or other parts of the screen) to authorize the external resource to access user data. This menu identifies the type of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, interactive client 104 adds the external resource to the list of authorized external resources and allows the external resource to access user data from interactive client 104. External resources are authorized by the interactive client 104 to access user data under the OAuth 2 framework.

[0096] Interactive client 104 controls the type of user data shared with external resources based on the type of authorized external resource. For example, it provides access to a first type of user data (e.g., two-dimensional avatars of users with or without different avatar characteristics) to external resources including full-scale applications (e.g., application 106). As another example, it provides access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar characteristics) to external resources including smaller versions of applications (e.g., web-based versions of applications). Avatar characteristics include different ways of customizing the appearance and feel of an avatar (e.g., different poses, facial features, clothing, etc.).

[0097] The advertising system 228 enables third parties to purchase advertisements to be presented to end users via the interactive client 104, and also handles the delivery and presentation of these advertisements.

[0098] Artificial intelligence and machine learning system 230 provides various services to different subsystems within interactive system 100. For example, artificial intelligence and machine learning system 230 operates in conjunction with image processing system 202 and camera device system 204 to analyze images and extract information, such as objects, text, or faces. The image processing system 202 or augmentation system 206 can then use this information to enhance, filter, or manipulate (e.g., apply AR effects) the images. Artificial intelligence and machine learning system 230 can also operate in conjunction with AR effect description system 232, as described below, to analyze images and generate descriptions of AR effects applied to or presented in these images.

[0099] Artificial intelligence and machine learning systems 230 can implement visual semantic machine learning models (also known as visual language models). As used herein, the term "visual semantic machine learning model" refers to a machine learning model or a combination of machine learning models that has the ability to process both visual (e.g., images or videos) and linguistic (e.g., text) data. Visual semantic machine learning models combine visual and semantic modalities to provide applications such as image captions and visual question answering (VQA). Examples of the use of visual semantic machine learning models are described below:

[0100] Image caption text addition: The visual semantic machine learning model receives the input image and generates a natural language text description or caption for the image.

[0101] VQA: Visual-Semantic Machine Learning Models receive images and prompts (e.g., questions related to images in natural language text format) and generate natural language text responses to the prompts based on the images.

[0102] Text-to-image synthesis: Visual semantic machine learning models receive a textual description of a scene or object and generate an image designed to match that description.

[0103] Semantic image retrieval: Visual semantic machine learning models receive text queries and retrieve one or more images that are considered to match (or be most relevant to) the text query based on semantic understanding.

[0104] Image-to-text translation: Visual semantic machine learning models receive images containing text (e.g., road signs or pages in a book), extract the text, and translate the text into another (natural) language.

[0105] Phrase localization: Visual semantic machine learning models perform object detection from input images and natural language phrases.

[0106] Examples of machine learning models that can provide such multimodal capabilities include models based on "BLIP" (Bootstaff Language Image Pre-training) strategies (e.g., BLIP and BLIP-2). For example, BLIP-2 is a scalable multimodal pre-training method that enables LLMs to ingest and understand images, thereby achieving image-to-text generation, VQA, and other functionalities. Another example of a machine learning model is a CLIP (Contrastive Language Image Pre-training) type model. More details about example machine learning models are provided below.

[0107] In some examples, artificial intelligence and machine learning systems 230 are used for training machine learning models. Training can involve fine-tuning, such as fine-tuning the visual semantic machine learning model to improve its performance in describing AR effects. For example, parameter-efficient fine-tuning processing can be used to fine-tune the visual semantic machine learning model to adapt its original "add descriptive text" skill, so that the visual semantic machine learning model "adds descriptive text" to AR effects appearing in an image, rather than describing the entire image or features unrelated to the AR effect.

[0108] The artificial intelligence and machine learning system 230 can also be used by the augmentation system 206 to generate augmented content and AR experiences, such as adding virtual objects or animations to real-world images. The communication system 208 and the messaging system 210 can use the artificial intelligence and machine learning system 230 to analyze communication patterns and provide insights into how users interact with each other, as well as to provide intelligent message classification and tagging, such as classifying messages based on sentiment or topic.

[0109] The artificial intelligence and machine learning system 230 can also provide chatbot functionality for message interactions 120 between user systems 102 and between user systems 102 and the interaction server system 110. The artificial intelligence and machine learning system 230 can provide various generative functions (e.g., allowing users to generate text, images, or video content based on prompts). The artificial intelligence and machine learning system 230 can work with the audio communication system 216 to provide speech recognition and natural language processing capabilities, enabling users to interact with the interaction system 100 using voice commands.

[0110] It may be desired to generate descriptions of AR effects. The AR effect description system 232 can perform various functions related to the generation and / or storage of descriptions of AR effects. For example, if the interactive system 100 supports a large number of AR effects (e.g., thousands or even millions of AR effects), the AR effect description system 232 can (e.g., by working with the image processing system 202) implement an automated pipeline for rendering each of the multiple AR effects on the appropriate content item.

[0111] AR effect description system 232 can operate in conjunction with artificial intelligence and machine learning system 230 to generate descriptions of target AR effects and store these descriptions for downstream use within the context of interactive system 100. In some examples, the description or portions thereof (e.g., words or phrases) can be tokenized for downstream use (e.g., for input into an embedding model).

[0112] Data Architecture

[0113] Figure 3 This is a schematic diagram illustrating a data structure 300 that can be stored in a database (e.g., database 128 of interactive server system 110) according to certain examples. Although the contents of database 128 are shown as including multiple tables, it should be understood that data can be stored in other types of data structures (e.g., object-oriented databases).

[0114] Database 128 includes message data stored in message table 304. For any given message, this message data includes at least message sender data, message receiver (or recipient) data, and payload. See below for reference. Figure 16 Further details are provided regarding information that can be included in the message and is contained within the message data stored in message table 304.

[0115] Entity table 306 stores entity data and (for example, links to entity diagram 308 and profile data 302). Entities for which records are maintained in entity table 306 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of the entity type, any entity for which the interactive server system 110 stores data can be an identifiable entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0116] Entity Graph 308 stores information about relationships and associations between entities. As an example only, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based. Some relationships between entities can be one-way, such as an individual user subscribing to digital content (e.g., a newspaper or other digital media channel or brand) for a business or publishing user. Other relationships can be two-way, such as the "friend" relationship between the various users of Interactive System 100.

[0117] Certain licenses and relationships can be attached to each relationship, and also to each direction of the relationship. For example, a two-way relationship (e.g., a friend relationship between individual users) may include authorization for the public disclosure of digital content items between the individual users, but certain restrictions or filters may be imposed on the public disclosure of these digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual user and a business user may impose varying degrees of restrictions on the public disclosure of digital content from the business user to the individual user, and may significantly limit or prevent the public disclosure of digital content from the individual user to the business user. As an example of an entity, a specific user may (e.g., through privacy settings) record certain restrictions in the records for that entity within entity table 306. Such privacy settings may be applied to all types of relationships in the context of the interaction system 100, or selectively applied to certain types of relationships.

[0118] Profile data 302 stores various types of profile data about a specific entity. Based on privacy settings specified by the specific entity, profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case of an individual, profile data 302 includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a set of such avatar representations) selected by the user. The specific user can then selectively include one or more of these avatar representations within the content of messages transmitted via the interaction system 100 and on a map interface displayed to other users by the interaction client 104. The set of avatar representations may include “status avatars,” which present a graphical representation of a status or activity that the user can choose to transmit at a specific time.

[0119] In the case that the entity is a group, in addition to the group name, members and various settings for the relevant group (e.g., notifications), the profile data 302 for the group may similarly include one or more avatars associated with the group.

[0120] Database 128 also stores augmentation data in augmentation table 310, including AR effects such as filters and overlays. The augmentation data is associated with and applied to videos (video data is stored in video table 312) and images (image data is stored in image table 314).

[0121] In some examples, filters include visual content displayed as an overlay on an image or video during presentation to the recipient user. Filters can be of various types, including user-selected filters from a set of filters presented to the sending user by the interactive client 104 while the sending user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, geolocation filters specific to nearby or particular locations can be presented by the interactive client 104 within the user interface based on geolocation information determined by the Global Positioning System (GPS) unit of the user system 102.

[0122] Another type of filter is a data filter, which can be selectively presented to the sending user by the interactive client 104 based on other inputs or information collected by the user system 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the sending user's current speed, the battery life of the user system 102, or the current time.

[0123] Other augmented data that can be stored within image table 314 includes, for example, AR content items corresponding to the application's "Lens" or AR experience. AR content items can include real-time effects that can be added to images or videos.

[0124] Collection table 316 stores data about collections of messages and associated image, video, or audio data, compiled into collections (e.g., stories or galleries). The creation of a specific collection can be initiated by a specific user (e.g., each user whose records are maintained in entity table 306). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. For this purpose, the user interface of interactive client 104 can include user-selectable icons to allow the sending user to add specific content to their personal story.

[0125] The collection can also constitute a "live story," which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic technologies. For example, a "live story" can constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and who are at a co-located event at a specific time can be presented with the option to contribute content to a specific live story, for example, via the user interface of interactive client 104. Live stories can be identified by interactive client 104 based on a user's location. The end result is a "live story" told from a community perspective.

[0126] Another type of content collection is called a "location story," which allows users of user system 102 located in a specific geographic location (e.g., on a college or university campus) to contribute to a specific collection. In some examples, contributions to a location story may employ secondary authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0127] As mentioned above, video table 312 stores video data, which in some examples is associated with messages for which records are maintained within message table 304. Similarly, image table 314 stores image data associated with messages whose message data is stored in entity table 306. Entity table 306 can associate various enhancements from enhancement table 310 with various images and videos stored in image table 314 and video table 312.

[0128] AR effect description table 318 stores descriptions of various AR effects supported by interactive system 100. For example, each AR effect may have a unique identifier (e.g., a code or name), where a natural language text description of the AR effect is stored in association with the identifier in AR effect description table 318. The descriptions facilitate searching. For example, when a user of user system 102 searches for keywords via interactive client 104, interactive client 104 can locate matching or related AR effects by examining the descriptions in AR effect description table 318. In some cases, the data in AR effect description table 318 can be indexed for easy searching and retrieval.

[0129] In some examples, the AR effect description system 232 of the interactive system 100 uses artificial intelligence and machine learning system 230 to automatically generate descriptions for AR effects (e.g., new AR effects generated by users of the augmentation creation system 214 and uploaded to the interactive system 100), and stores these descriptions in an AR effect description table 318 for downstream use. The AR effect description table 318 may also include other AR effect metadata, such as AR effect tags, categories, groups, or rankings. For example, the AR effect description system 232 may use a visual semantic machine learning model to generate AR effect descriptions and also identify the categories of the AR effects (e.g., sports, birthday, or dog), and then store the categories in the AR effect description table 318 in association with the AR effect's identifier.

[0130] Figure 4 This is shown based on some examples. Figure 1 and Figure 2 Figure 400 shows the components of the AR effect description system 232 of the interactive system 100 and the artificial intelligence and machine learning system 230.

[0131] The AR effect description system 232 and the artificial intelligence and machine learning system 230 can communicate with each other for one or more purposes (e.g., using the machine learning capabilities of the AR effect description system 232 to generate AR effect descriptions). The AR effect description system 232 and the artificial intelligence and machine learning system 230 can communicate with one or more storage components (e.g., Figure 1 It communicates with the database (128 or another storage type) for data storage and retrieval.

[0132] In some examples, Figure 4 At least some of the components shown are configured to communicate with each other to implement the aspects described herein. One or more of the components described herein may be implemented using hardware (e.g., one or more processors of one or more machines) or a combination of hardware and software. For example, a component described herein may be implemented by a processor configured to perform the operations described herein for that component. Furthermore, two or more of these components may be combined into a single component, or the functionality described herein for a single component may be further divided among multiple components. Additionally, according to various examples, the components described herein may be implemented using a single machine, database, or device, or may be distributed across multiple machines, databases, or devices.

[0133] The AR effect description system 232 is shown as including a rendering component 402, a video collection component 404, a frame extraction component 406, a stitching component 408, a description generation component 410, and a data storage component 412. The artificial intelligence and machine learning system 230 is shown as including a fine-tuning component 414, a baseline model 416, and a fine-tuned model 418.

[0134] Rendering component 402 is responsible for rendering AR effects on content items such as images or videos. For example, rendering component 402 can access a first video or base video that does not include a specific AR effect, and then apply the AR effect to the base video to produce a second video or enhanced video. For example, the first video could be a video of a person waving at the camera of a recording device, and the AR effect could be a color filter that changes the color of the person's hair (e.g., from brown to blue), such that the enhanced video shows the person waving at the camera, but with a different hair color than in the base video. Rendering component 402 can similarly apply AR effects to still images to produce enhanced images.

[0135] Rendering component 402 can work with image processing system 202, for example, to perform object detection and apply AR effect algorithms to content items. Rendering component 402 can store enhanced content items (e.g., content items or links / paths to them can be stored in database 128 along with corresponding base content items or links / paths to them). Enhanced content items can be used by artificial intelligence and machine learning system 230 during training (e.g., fine-tuning) and / or for inference, as further described below.

[0136] Rendering component 402 can implement or access automated rendering services (e.g., automated AR effects rendering services), which may be referred to as "lens rendering services" or "filter rendering services." Such services can provide a pipeline of suitable content, including enhanced video or enhanced images. In some examples, the service includes an application configured to render specific AR effects on a base video (which may also be referred to as a substrate video). In other words, the rendering service provides an automated service capable of supplying video or image content on which desired AR effects can be applied. The rendering service can retrieve videos from a database 128 or a library in another storage component. For example, the rendering service can access the base video and individually render (by way of example only) 50, 100, or 1500 AR effects onto the base video to obtain 50, 100, or 1500 enhanced videos for downstream use.

[0137] The video collection component 404 can be configured to (e.g., from database 128) retrieve or collect pairs of videos. As described above, a video pair may include a base video and its corresponding enhanced video.

[0138] Frame extraction component 406 is responsible for extracting corresponding frames from the base video and its associated enhanced video. Frame extraction component 406 can be configured to extract individual frames from digital video files or streams.

[0139] For example, and as mentioned above, the base video could depict a person waving at the camera, and its associated enhanced video could depict a person with different hair colors produced by AR effects. The frame extraction component 406 can preprocess the video by extracting corresponding frames from the base video and the enhanced video to allow for useful comparisons of the visual features therein.

[0140] The corresponding frames can be those with the same (or substantially the same) timestamps or the same (or substantially the same) sequence positions between the base video and the enhanced video. As mentioned, minor differences (e.g., due to time drift) can be acceptable.

[0141] In this way, objects, environment, and positions in the two extracted frames can be essentially aligned, except for the AR effects (which are not present in the base video) and the changes caused by the AR effects. The frame extracted from the base video is referred to below as the "first image," and the frame extracted from the augmented video is referred to below as the "second image."

[0142] These paired first and second images can be used by the artificial intelligence and machine learning system 230 for both training and inference / prediction purposes. When the first and second images are provided to the machine learning model for inference / prediction, they are referred to as the "first input image" and the "second input image," respectively, and if the first and second images are used to train (e.g., fine-tune) the machine learning model, they are referred to as the "first training image" and the "second training image," respectively.

[0143] In some examples, frames are selected based on the differences between the base video and the enhanced video. For instance, frame extraction component 406 can automatically select a pair of frames (e.g., a first image and a second image) that includes a frame from the base video and a temporally corresponding frame from the enhanced video, where there is a high degree of pixel difference between the two frames. In some cases, the pair of frames that provides the highest degree of pixel difference can be selected.

[0144] To determine the difference between a pair of frames (e.g., a first image and a second image), the frame extraction component 406 can calculate a pixel-wise loss between the frames. In this way, the frame extraction component 406 can be able to evaluate multiple pairs and find a frame index that provides a contrasting image useful in representing related AR effects (or aspects thereof).

[0145] In some examples, the first and second images are stitched or combined before they are processed for AR effect description generation. The stitching component 408 can access the two corresponding images and stitch them together, for example, based on a predetermined spatial arrangement. For example, the first and second images can be joined horizontally so that they are positioned side by side in the stitched image, or vertically so that one image is on top of the other.

[0146] Designing and / or implementing systems or components that can understand AR effects (e.g., media overlays, filters, or AR experiences) can be a challenging task. As mentioned above, to obtain a useful description of an AR effect, a first and a second image can be fed into a visual semantic machine learning model. In response, the visual semantic machine learning model can provide an output (e.g., a natural language text response) describing one or more features of the AR effect.

[0147] The fine-tuning component 414 of the artificial intelligence and machine learning system 230 can be configured to fine-tune a pre-trained model (e.g., a pre-trained visual semantic machine learning model, such as a BLIP-2-based model). Such a pre-trained model can be referred to as a baseline model, and example baseline model 416 is shown in... Figure 4 As shown in the figure. In some examples, the fine-tuning component 414 uses fine-tuning techniques such as parameter-efficient fine-tuning techniques to adapt the baseline model 416 to generate or obtain a fine-tuned model 418.

[0148] In some examples, and as described in more detail below, the LoRA fine-tuning method can be implemented by injecting additional low-rank weight matrices into the attention layer of the baseline model 416. The fine-tuning component 414 then updates only these matrices while keeping the original weights of the baseline model 416 fixed. This selective updating prevents interference with the general knowledge in the pre-trained baseline model 416 while adapting the baseline model 416 to focus on AR effects.

[0149] The fine-tuned model 418 is an adapted model generated through the operation of the fine-tuning component 414. In some examples, the fine-tuned model 418 contains the same components as the baseline model 416, but has been specialized via an injected weight matrix to focus on describing the AR effect. When the fine-tuned model 418 receives an input image pair illustrating the AR effect, it can isolate the AR effect and generate a textual description of the AR effect as output. In some examples, the general add explanatory text feature of the baseline model 416 is fine-tuned to "add explanatory text" only for the AR effect, thus providing AR effect-specific explanatory text.

[0150] The description generation component 410 of the AR effect description system 232 is responsible for generating, selecting, aggregating, enhancing, or modifying descriptions of specific AR effects. In some cases, the description generation component 410 may select output from the baseline model 416 and directly use or apply such output as a description of the relevant AR effect. In other cases, the description generation component 410 may generate new or modified descriptions based on the output from the baseline model 416. In some examples, the description generation component 410 uses only a subset of the outputs generated by the baseline model 416 (e.g., by ignoring outputs identified as irrelevant or potentially irrelevant). In some examples, as further described below, the description generation component 410 (e.g., using a machine learning model such as the CLIP model) scores multiple outputs and selects a subset of outputs based on the scores.

[0151] Data storage component 412 can be as follows Figure 3Data related to AR effects is stored as shown (e.g., in AR effect description table 318 of database 128). For example, data storage component 412 can store the output or response received from the fine-tuned model 418 in database 128. Data storage component 412 can store descriptions of AR effects selected, confirmed, or generated by description generation component 410 in association with identifiers of AR effects. Data storage component 412 can store such data for subsequent retrieval. For example, data storage component 412 can store data in the following format... Figure 3 In the AR effect description table 318, this format facilitates retrieval in response to AR effect search queries originating from the interactive client 104 operating within the interactive system 100, as shown below. Figure 12 and Figure 13 As described.

[0152] Figure 5 This is a flowchart illustrating a method 500 for fine-tuning a visual semantic machine learning model based on some examples. The operations in method 500 can be used by the interactive system 100 as described above. Figures 1 to 4 The components described (e.g., parts, modules, systems, or engines) are used for execution. Therefore, reference is made to the relevant information through examples rather than limitations. Figures 1 to 4 The components and interaction system 100 are described to describe the method 500.

[0153] Note that, although the interactive system 100 is described... Figure 5 The operation is permissible, but it should be understood that at least some of the operations can be deployed on various other hardware configurations or performed by similar components residing elsewhere. (See also...) Figure 6 and Figure 7 Further description Figure 5 Method 500 Figure 6 and Figure 7 The visual semantic machine learning model and various aspects of model fine-tuning are shown separately.

[0154] Method 500 begins at the start loop element 502 and proceeds to operation 504, where the AR effect description system 232 generates training images. For example, rendering component 402 can render the enhanced video as described above, wherein video collection component 404 and frame extraction component 406 are used to identify corresponding image frame pairs from the enhanced video and its base video. As mentioned, frame extraction component 406 can select a first training image and a corresponding second training image (rather than other pairs from the base video and enhanced video) based on detected differences or contrasts (e.g., high pixel dissimilarity).

[0155] As discussed elsewhere, the stitching component 408 can stitch together pairs of first and second images to reflect the AR effect. In this way, each training sample can simulate a specific "before and after" effect supported by the interactive system 100. See below. Figures 9 to 11 A more detailed example of splicing. Although about Figures 9 to 11 The description refers to the input images used for inference, but similar stitching can be applied to generate or modify training data.

[0156] In some examples, training samples are obtained from real data in the interactive system 100, thereby reducing or eliminating the need to generate samples specifically for training purposes.

[0157] At operation 506, the AR effect description system 232 obtains a training description for each training sample. In other words, each training sample includes not only the stitched image but also a description of the AR effect applied to generate a second image in the stitched image. The training description can be obtained, for example, through manual labeling. For example, the training description can be a text description of the relevant AR effect (e.g., "add a dog nose and dog ears to a face" or "change the color of a person's hair to blue"). Then, at operation 508, the training images and their corresponding descriptions are stored as training data. For example, the training data (or a link / path to it) can be stored in... Figure 1 In database 128. As mentioned below, training data may include data to be used for training, as well as separate data to be used for validation.

[0158] As a non-limiting example, a total of 100,000 samples can be used, where the descriptions in each sample are generated by human labelers. Therefore, these samples can be referred to as labeled data. In some examples, the initial sample set is filtered to obtain the final dataset. For example, a subset of the initial set of 100,000 samples can be selected, which contains the word "face" in the descriptions. This could, for example, provide a final set of 35,000 samples specifically selected to help optimize a visual semantic machine learning model to describe facial AR effects. The final dataset can be split into a training dataset and a validation dataset (e.g., according to an "80 / 20" split).

[0159] Method 500 proceeds to operation 510, where a baseline model is selected. For example, it can be used... Figure 4 The baseline model 416. The baseline model 416 can be a pre-trained visual semantic machine learning model.

[0160] Figure 6An example of a visual semantic machine learning model 602 is shown. As mentioned, BLIP and BLIP-2 are non-limiting examples of visual semantic machine learning models with multimodal capabilities. Visual semantic machine learning model 602 is based on the BLIP-2 framework. BLIP-2 is described below to illustrate certain aspects that can be applied to implement visual semantic machine learning models. However, it should be understood that other types of visual semantic machine learning models can be used to implement the techniques described herein. For example, other multimodal models incorporating LLM can be used.

[0161] BLIP-2 is a scalable multimodal pre-training method that enables LLMs to ingest and understand images. For example, given an input image and a cue (e.g., a text instruction), a BLIP-2-based model can generate a natural language response based on the cue. For instance, the model might receive an image of the Great Wall of China and the cue "Tell me some history about this place." The BLIP-2-based model could respond by providing a textual description indicating when and why the Great Wall was built. As another example, the model might receive an image of a dog and the cue "Describe this animal." The model could respond by stating that the animal is a dog and describing its characteristics. As yet another example, a BLIP-2-based model can provide an image description or explanatory text based solely on the input image (e.g., without accompanying cue).

[0162] Figure 6 The visual semantic machine learning model 602 utilizes a frozen visual model in the form of an image encoder 604 and a frozen language model in the form of an LLM 606. In the context of an LLM, the LLM may not have seen any images during its natural language pre-training, thus generating a so-called "visual-language modality gap." Bridging this gap can be challenging, especially when the LLM remains frozen. BLIP-2 provides a method for bridging this gap, for example, by enabling the LLM to understand visual content using a query transformer (also known as a Q-former), which is pre-trained using a two-stage pre-training strategy. The visual semantic machine learning model 602 includes a Q-former 608 between the image encoder 604 and the LLM 606.

[0163] Therefore, the Q-former 608 is connected to both the visual model (e.g., image encoder 604) and the language model (e.g., LLM 606). In some examples, the Q-former 608 is the only part of the visual semantic machine learning model 602 that can be trained as part of BLIP-2 processing (the other parts have been pre-trained and remain frozen during the training of the Q-former 608). The Q-former 608 may contain two sub-components:

[0164] An image transformer interacts with (frozen) image encoder 604 to extract visual features. A Q-former 608 can extract information from the Q-former 608 using a set of trainable query vectors (e.g., a fixed number of output features, independent of the input image resolution).

[0165] A text transformer that can function as both a text encoder and a text decoder.

[0166] The Q-former 608 can be trained to extract the most informative visual representation of text. In the BLIP-2 strategy, the following three pre-training objectives can be optimized:

[0167] Image-Text Contrast Learning (ITC). This type of learning is used to learn how to align image and text representations to maximize their mutual information.

[0168] Image-based text generation (ITG). This type of learning is used to teach a Q-former how to generate text.

[0169] Image-Text Matching (ITM). This type of learning is used to teach fine-grained alignment between image and text representations.

[0170] The first pre-training phase can be referred to as visual and language representation learning. In this phase, the Q-former 608 is connected to the frozen visual model (e.g., image encoder 604) and pre-trained with image-text pairs. The Q-former 608 learns to extract image features most relevant to the corresponding text.

[0171] The second pre-training phase can be termed vision-to-language generative learning. In this phase, the output of the Q-former 608 is connected to a frozen language model (e.g., LLM 606). The Q-former 608 is trained so that its output features can be interpreted by the LLM to generate the corresponding text. The LLM used in the BLIP-2 model can include either a decoder-based LLM or an encoder-based LLM.

[0172] Typically, after pre-training, a Q-former (e.g., Q-former 608) can effectively act as a bridge between the visual and language models, bridging the visual-language modality gap. During inference, the visual-semantic machine learning model 602 can receive images and cues (e.g., text instructions). The visual-semantic machine learning model 602 can append text instructions to the output of the Q-former 608 as input to the LLM. The visual-semantic machine learning model 602 can also receive only images. In some examples, when the image is not accompanied by cues, the visual-semantic machine learning model 602 will generate descriptive text based on the image content represented as in the output of the Q-former 608.

[0173] The output of Q-former 608 can be linearly projected to match the size of the text embedding in LLM 606. This allows LLM 606 to embed visual information extracted from Q-former 608.

[0174] Therefore, a visual model (e.g., image encoder 604) can extract features from an image, where intermediate components (e.g., Q-former 608) select relevant information while filtering out information irrelevant to the LLM (e.g., text output 610). Such as Figure 6 The visual semantic machine learning model 602 can be used for various purposes, such as VQA, image text retrieval, or image caption text addition. For example... Figure 6 As shown, when image caption text is added, as mentioned above, the visual semantic machine learning model 602 can generate text output 610 based on the image input 612 to describe the visual features of the image input 612.

[0175] Return to reference Figure 5 At operation 512, parameter-efficient fine-tuning is used to fine-tune the baseline model 416 on the relevant training data (e.g., Figure 6 The visual semantic machine learning model 602). The parameter-efficient fine-tuning process can be used to fine-tune the baseline model 416 to essentially “ignore” content in the input image that is irrelevant to the AR effect, and thus generate AR effect-specific descriptions or explanatory text.

[0176] During fine-tuning, the baseline model 416 can be trained on the training samples described above (e.g., each stitched image showing the enhanced "before and after" effect, along with training descriptions or labels). This improves the baseline model 416's ability to add descriptive text when it comes to describing AR effects. Furthermore, the parameter-efficient fine-tuning process provides a lightweight mechanism that allows for relatively fast fine-tuning with minimal resource requirements. An example of such a process is LoRA, where the attention weights of the machine learning model are updated while other weights remain frozen. See below. Figure 7To describe LoRA.

[0177] Once fine-tuning is complete, the fine-tuned model 418 (e.g., an adapted version of the visual semantic machine learning model 602) can be evaluated. For example, the performance of the fine-tuned model 418 can be compared to the performance of the baseline model 416 or to descriptions provided by human labelers. If the fine-tuned model 418 is properly fine-tuned, it can perform better than the baseline model 416 in terms of focused AR effects and can also provide faster and more consistent output than human labelers.

[0178] In some examples, fine-tuning is performed only on the Q-former 608, while keeping the image encoder 604 and LLM 606 fixed. While a baseline version of the Q-former 608 can provide the LLM 606 with an output (e.g., embedding) focused on both the AR effect and other image content (such as other artificial features or background), a fine-tuned version of the Q-former 608 can, as a result of the fine-tuning process, provide the LLM 606 with an output more specifically focused on the AR effect. This allows the LLM 606 to generate textual output that specifically describes the AR effect present in the relevant input image (in some examples, it can show the visual transformation caused by the AR effect to facilitate further analysis by the visual semantic machine learning model 602). Below... Figure 14 and Figure 15 The description provides more information about machine learning processes, including, for example, model evaluation.

[0179] Deploy (operation 514) the finely tuned model 418 (e.g., an adapted version of the visual semantic machine learning model 602) to generate an AR effect description for the target AR effect. Figure 5 Method 500 ends at the end of the loop at element 516.

[0180] Now refer to Figure 7 LoRA Processing 700 is illustrated as a non-limiting example of a parameter-efficient fine-tuning technique. In some cases, fine-tuning large models is expensive and time-consuming. LoRA Processing 700 involves freezing pre-trained model weights and injecting trainable layers in the form of rank factorization matrices. In the case of a transformer architecture, these layers can be injected into some or all of the transformer blocks. This reduces the number of trainable parameters and thus reduces the GPU memory requirements for the fine-tuning task.

[0181] When using LoRA to process 700, the model's original weights are frozen during fine-tuning. Instead, modifications are applied to individual weight sets, and their new values ​​are added to the original parameters.

[0182] Now, let's get to the specifics. Figure 7LoRA processing 700 can be used to adapt pre-trained weights (e.g., self-attention weights of transformers). Figure 7 The input 702 to the layer is represented as X, and the original pre-trained (frozen) weight matrix 704 is represented as W, which are defined as follows:

[0183]

[0184] Initially, before fine-tuning, to obtain the final output 714 of the layer (denoted as H), the model performs H = WX as its forward pass. The update or adaptation matrix to be learned can be denoted as... Represented by low-rank decomposition. This allows for constraints on it:

[0185]

[0186] in,

[0187] And among them, rank

[0188] exist Figure 7 In the diagram, A and B are shown as low-rank matrices 706 and 708, respectively. W (weight matrix 704) and (Low-rank matrix 706 and low-rank matrix 708) are both multiplied by the same input 702. The output vector is summed coordinate by coordinate. The new forward pass becomes:

[0189]

[0190] exist Figure 7 In the diagram, WX is shown as output 710, and BAX is shown as output 712. The final trained forward pass can be expressed as follows (merging the pre-trained W with the low-rank update BA to obtain the final output 714):

[0191]

[0192] To illustrate parameter efficiency, we can assume an original 100×100 matrix. This matrix represents the weights of the pre-trained layers (e.g., a portion of the model's attention weights). LoRA processing 700 can be used through two low-rank matrices (e.g., ... Figure 7The original matrix is ​​approximated by low-rank matrices 706 and 708 (described in the diagram). For example, low-rank matrix 708 can have a size of 100×5, and low-rank matrix 706 can have a size of 5×100. When the number of parameters in the original matrix is ​​10000, the total number of parameters in low-rank matrices 706 and 708 is 1000. This allows the layer to be fine-tuned with only 1000 parameters instead of the original 10000 parameters, making fine-tuning faster and more efficient.

[0193] In some examples, W is frozen and does not receive gradient updates during training, while A and B contain trainable parameters and are updated. In some examples, random Gaussian initialization can be used for A, and B can be zero, such that when LoRA processes 700, The result is zero. When deployed in production, the total weight matrix W + BA can be calculated and stored, and inference can be performed as usual. In some examples, a merging ratio α can be applied to replace BA with αBA.

[0194] By using parameter-efficient fine-tuning techniques (such as LoRA), visual semantic machine learning models (such as Visual Semantic Machine Learning Model 602) can be efficiently fine-tuned to alter their ability to add descriptive text to describe AR effects rather than the entire image content. In some examples, Capture the modifications required for the desired task with significantly fewer parameters.

[0195] This approach eliminates the need (and the computational resources required) to add custom cues to the input image when attempting to obtain a more focused description of AR effects (rather than a holistic description) from a visual semantic machine learning model. Fine-tuned models can be applied across a large number of AR effects (e.g., Figure 4 The finely tuned model 418 is used to capture an accurate description in a consistent and / or uniform manner.

[0196] Fine-tuning can be efficient in terms of computational resources and time by freezing most of the pre-trained weights and updating only certain layers (such as attention layers). For example, Figure 7 The trainable matrices A and B can be inserted into the multi-head self-attention layer of the baseline model 416 containing the transformer block, while keeping other parameters frozen.

[0197] Figure 8 This is a flowchart illustrating a method 800 suitable for generating AR effects, based on some examples. The operations in method 800 can be used by the interactive system 100 as described above. Figures 1 to 4 The components described (e.g., parts, modules, systems, or engines) are used for execution. Therefore, reference is made to the relevant information through examples rather than limitations. Figures 1 to 4The components and interaction system 100 are described to describe the method 800.

[0198] Note that, although the interactive system 100 is described... Figure 8 The operation is permissible, but it should be understood that at least some of the operations can be deployed on various other hardware configurations or performed by similar components residing elsewhere. (See also...) Figures 9 to 11 The example image shown further illustrates method 800.

[0199] Method 800 begins at the start of the loop element 802 and proceeds to operation 804, where AR effects are applied to the base video to obtain an enhanced video. Figure 8 China (and) Figure 10 and Figure 11 (See together), using the "dog face" filter as a non-limiting example of an AR effect. For example, the base video could be a video depicting a person (including a person's face), but without any AR effects applied to it. Then, the AR effect description system 232 (see...) Figure 4 The rendering component 402 applies AR effects to add dog features to a human face (see [link]). Figure 10 (This will be discussed below).

[0200] At operation 806, video collection component 404 retrieves the base video and enhanced video, and frame extraction component 406 extracts a pair of corresponding frames (e.g., time-corresponding frames as described above). Thus, a first input image and a second input image are obtained by frame extraction component 406. As mentioned, frame extraction component 406 may select the first input image and the second input image (rather than other pairs from the base video and enhanced video) based on detected differences or contrasts (e.g., high pixel dissimilarity).

[0201] For example, the first input image comes from a base video and shows a person's "normal" (e.g., unenhanced) face, while the second input image comes from an enhanced video and shows the same content as the first input image, except for dog features superimposed on the person's face by the rendering component 402. The rendering component 402 can operate in conjunction with the image processing system 202 to process the base video such that the enhanced video includes a "dog face" filter.

[0202] Figure 9 An example of the first image 900 is shown, and Figure 10 An example of the second image 1000 is shown, which includes an AR effect 1002 in the form of an example "dog face" filter. The first image 900 and the second image 1000 can be used (e.g., together with the training description) as part of the training data, or as in Figure 8 It is used during inference in the same way as in method 800.

[0203] Therefore, the first image 900 and the second image 1000 are used as examples of the first and second input images to describe certain aspects of method 800 below. However, it should be understood that many other types of AR effects or images can be used in other examples of the technique described herein.

[0204] Method 800 proceeds to operation 808, where the stitching component 408 generates a stitched image including the first image 900 and the second image 1000. Figure 11 An example of a stitched image 1100 is shown, in which the first image 900 and the second image 1000 are joined together in a horizontal stitching manner.

[0205] Specifically, in Figure 11 In this configuration, the first image 900 is located to the left of the stitched image 1100, and the second image 1000 is located to the right of the stitched image 1100 to provide a left-to-right transformation 1102, as will be further described below. The stitched image 1100 is a non-limiting example used to describe certain aspects of method 800, and it will be understood that other types of stitching (e.g., vertical stitching) may be used in other examples of the techniques described herein. Furthermore, in some examples, the stitched image may include more than two images, such as a series of three frames from a base video and a corresponding (e.g., temporally corresponding) series of three frames from an enhanced video.

[0206] Now refer to Figure 8 The operation 810 then provides the stitched image 1100 (e.g., automatically provided by the AR effect description system 232) to a visual semantic machine learning model, such as a fine-tuned model 418 (which may be...). Figure 6 (A fine-tuned version of the visual semantic machine learning model 602). The stitched image 1100 shows the AR effect 1002, which in this context can be referred to as the "target AR effect".

[0207] As described above, the visual semantic machine learning model is trained and fine-tuned in some examples to describe the visual effects of the AR effect 1002 in question. For example, the fine-tuned model 418 can be adapted to analyze the left-to-right transformation 1102 (based on its fine-tuning on “before and after” images arranged similarly with corresponding explanatory text) and generate explanatory text that specifically describes the AR effect 1002 (in other words, performing an AR effect-specific explanatory text addition operation).

[0208] Therefore, in some examples, only the stitched image 1100 is provided to the visual semantic machine learning model (e.g., without any explanatory text instructions or explanatory text prompts), and no explicit instructions (e.g., text prompts) are required.

[0209] After the stitched image 1100 is provided to a visual semantic machine learning model (e.g., a fine-tuned model 418 of the artificial intelligence and machine learning system 230), at operation 812, the description generation component 410 of the AR effect description system 232 receives the output from the visual semantic machine learning model. This output describes at least one feature of the AR effect 1002, such as its visual effect. For example, the output from the visual semantic machine learning model could be: "Add a dog nose, dog ears, and a dog tongue to a human face."

[0210] Therefore, visual semantic machine learning models automatically perform visual feature extraction and present natural language descriptions of these features for downstream use. For example, see below. Figure 12 and Figure 13 Describe downstream usage in the search in more detail.

[0211] In this way, the ability of pre-trained visual semantic machine learning models to add descriptive text (especially after fine-tuning) can be used to automate and / or improve the generation of AR effect descriptions. Figure 1 In the interactive system 100, the output from the baseline model 416 can be stored (e.g., stored in...). Figure 3 The AR effect description is in Table 318 or in a table storing the "original" or "preliminary" description.

[0212] exist Figure 8 At operation 814 of method 800, the description generation component 410 generates or selects a description of the relevant AR effect. As mentioned, the description generation component 410 can directly use the output of the visual semantic machine learning model as the description, or it can modify the output.

[0213] For example, AR effect description system 232 can obtain multiple outputs related to the same AR effect (e.g., multiple different stitched images can be fed into a fine-tuned model 418, all of which have the same target AR effect), and description generation component 410 can aggregate the outputs or select from the outputs to obtain a final description. In some examples, description generation component 410 can utilize the machine learning model of artificial intelligence and machine learning system 230 to summarize, filter, and / or adjust the outputs (or multiple outputs) of fine-tuned model 418, or generate a set of keywords or tags describing the AR effect based on the outputs (or multiple outputs) of fine-tuned model 418.

[0214] In some examples, the description generation component 410 can automatically evaluate the quality or relevance of the response / output to determine whether to include the response in the final description or whether to consider the response when generating the final description. CLIP is an example of a neural network model that can be used for this purpose. Given an image and text description (e.g., an augmented video frame and one of the responses generated by the fine-tuned model 418), a CLIP-based model can determine how closely they match by computing a similarity score in a shared embedding space. The resulting score can be viewed as a measure of the relevance between a given image and text.

[0215] Therefore, the description generation component 410 can generate a relevance or similarity score, such as a CLIP score, for (or for each) output. A high score may indicate that the output is relevant and should be included in the final description or considered when generating the final description. Conversely, a low score may indicate a poor match or irrelevance. The description generation component 410 may, for example, include or consider all outputs with scores above a certain threshold, or include or consider a predetermined number of outputs with the highest scores.

[0216] In some examples, the description of generating component 410 utilizes an affinity score, which can be calculated as follows:

[0217] A text encoder using the CLIP model encodes the relevant response into an embedding vector.

[0218] The image encoder using the CLIP model also encodes the relevant images (e.g., augmented frames) into the embedding vector.

[0219] Then, the cosine similarity between the text and the image embedding is calculated to obtain a similarity score (e.g., between -1 and 1).

[0220] The cosine similarity is an "affinity score" that reflects the degree of matching between text and image content.

[0221] Then, by taking, for example, max(0,100) cos_sim(text_embedding, image_embedding)) limits the score to between 0 and 100.

[0222] Once the description is generated, the data storage component 412 stores the description of the target AR effect in association with the target AR effect (e.g., using the identifier of the target AR effect in the AR effect description table 318) (operation 816). This allows the interactive system 100 to automatically generate and store descriptions indicating "what" each AR effect is about. In this way, the descriptions can be used for downstream purposes, such as for search or ranking purposes. In some examples, the AR effect descriptions are used to index the data in the AR effect description table 318 to facilitate searching and retrieval.

[0223] For example, a user can use the interactive client 104 to execute... Figure 1 Use mobile devices to search 114 Figure 10 AR effect 1002 (“dog face” filter). Users can input search queries (such as “dog face”, “dog tongue”, etc.) through interactive client 104, and then match the search queries with the descriptions of AR effect 1002 stored in AR effect description table 318 (such as: “add dog nose, dog ears and dog tongue to a person’s face”).

[0224] In response to an AR effect 1002 matching a search query, the interactive client 104 can render image data at the mobile device 114 and apply the AR effect 1002 to the image data. In some cases, the AR effect 1002 is rendered and applied only after receiving user input confirming the selection of the AR effect 1002 (e.g., selecting from multiple displayed "matches").

[0225] At operation 818, AR effect 1002 can be applied to image data. In some cases, AR effect 1002 is applied in real time by a computing device associated with interactive system 100 (e.g., applied to objects presented in a camera feed interface of interactive client 104 presented on mobile device 114), thereby allowing the user to capture images or videos including AR effect 1002. In other cases, AR effect 1002 can be applied to previously captured content loaded from memory. Method 800 ends at the end loop element 820.

[0226] Note that, as provided via horizontal splicing Figure 11 The left-to-right transformation 1102 is a non-limiting example. For example, a right-to-left transformation can be provided by stitching images together, such that the first image (e.g., the image before enhancement) is on the right and the second image (e.g., the image after enhancement) is on the left. A top-to-bottom or bottom-to-top transformation can be provided via vertical stitching. In each case, the same predetermined spatial arrangement can be used during training (e.g., fine-tuning of the baseline model 416) and inference (e.g., generating descriptions for new, unseen AR effects) to improve model performance.

[0227] Although Figure 5 Method 500 and Figure 8 Method 800 involves stitching images (e.g., stitching image 1100), but in other examples, images may be fed to the machine learning model individually rather than in a stitched form. For example, in some cases, the first and second images may be provided separately to a visual semantic machine learning model, which then provides output to describe the observed visual effects.

[0228] As mentioned above, in some cases, it may be desirable to use more than two images. For example, in the case of a specific AR effect (e.g., an AR experience), different visual effects may be observed at different times (e.g., different stages of the AR experience). In this case, multiple pairs of images can be extracted from the base video and the augmented video separately.

[0229] For example, Figure 4 The frame extraction component 406 can extract a first image pair corresponding to a first time point in the two videos, a second image pair corresponding to a second time point in the two videos, and a third image pair corresponding to a third time point in the two videos, wherein each image pair shows different or varying features of the AR effect. A visual semantic machine learning model (e.g., a finely tuned model 418) can then be trained to describe the visual effects observed at these different stages, thereby allowing the generation of a comprehensive description of the AR effect.

[0230] In some examples, the techniques described in this paper can improve the functionality of content moderation systems by providing more accurate, detailed, or comprehensive descriptions of AR effects in an automated manner. Content moderation can be simplified, for example, by reducing the need for manual review of AR effects.

[0231] For example, return to reference Figure 2 The enhanced creation system 214 enables developers or content creators to create new AR effects and publish them to the interactive system 100. However, it may be desirable to review such new AR effects for offensive or harmful content. In some cases, the AR effect description system 232 can automatically determine that an AR effect contains or produces offensive or harmful content based on its description, and accordingly mark the AR effect in the AR effect description table 318. For example, the AR effect description system 232 can detect potentially offensive or harmful words, context, or emotions in the description of an AR effect and mark the associated AR effect. In response to the detection of this mark, the interactive system 100 may then prevent the AR effect from being used on the interactive client 104, or may automatically impose usage restrictions (e.g., age restrictions).

[0232] As described elsewhere, once an AR effect description is generated, the data storage component 412 can store the description in association with the AR effect. Table 1 below includes examples of such descriptions, which can be generated, selected, and / or aggregated, for example using the techniques described herein, and stored in AR effect description table 318 for use in the interactive system 100. The example provided is a panda-related AR effect and is provided for illustrative purposes only.

[0233]

[0234] Table 1: Examples of AR Effect Descriptions

[0235] For example, referring to the AR effect identified by AR effect ID 23101 in Table 1, this AR effect (through the automation techniques described herein) is described as adding pandas and leaves to an image of a person. Based on some examples, Figure 12 The first image 1200 is shown, and Figure 13 A second image 1300 is shown, including AR effect 1302 identified by ID 23101. The first image 1200 may be from a base video and shows an image of a "normally captured" (e.g., unenhanced) person, while the second image 1300 may be from an enhanced video and show the same content as the first image 1200, except (e.g., as shown by...). Figure 4 The rendering component 402 applies panda-related AR effects 1302.

[0236] Users can use the interactive client 104 to execute Figure 1 Mobile device 114 is used to search for AR effects related to pandas. For example, a user can enter the following search query: "Panda in a bamboo forest." Then, interactive client 104 can match the search query with the most relevant descriptions in AR effect description table 318 (e.g., one or more descriptions in Table 1). Interactive client 104 can identify the AR effect ID associated with each relevant description to display the relevant AR effect.

[0237] In some examples, the interactive client 104 may obtain a ranking of relevant AR effects (e.g., based on the degree of match between a search query and the corresponding description in the AR effect description table 318) from the augmentation system 206. The interactive client 104 may present relevant AR effects to the user based on the relevant ranking of relevant AR effects (e.g., some or all of the AR effects in Table 1), thereby allowing the user to select one of the AR effects to apply to a specific situation.

[0238] In response to the matching of AR effect 1302 with a search query and the user's selection of it from multiple options, interactive client 104 can enable the presentation of image data at mobile device 114 and apply AR effect 1302 to the image data, as described above.

[0239] Therefore, the techniques described in this paper provide useful metadata describing specific AR effects to facilitate applications such as searching for AR effects, generating AR effect recommendations, or ranking AR effects based on relevance. AR effect descriptions generated by fine-tuned visual semantic machine learning models can be of higher quality, more consistent, or more uniform than AR effect descriptions prepared by other labelers, and can be generated quickly and at scale.

[0240] Machine Learning Examples

[0241] Figure 14 This is a flowchart depicting a machine learning pipeline 1400 based on some examples. The machine learning pipeline 1400 can be used to generate trained models, for example... Figure 15 The trained machine learning program 1502 is shown in Figure 1500.

[0242] In a broad sense, machine learning can involve automatically learning patterns and relationships in data using computer algorithms, potentially without explicit programming. Machine learning algorithms can be categorized into three main types: supervised learning, unsupervised learning, and reinforcement learning.

[0243] Supervised learning involves training a model using labeled data to predict outputs for new, unseen inputs. Examples of supervised learning algorithms can include linear regression, decision trees, and neural networks.

[0244] Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships within the data. Examples of unsupervised learning algorithms can include clustering, principal component analysis, and generative models (such as autoencoders).

[0245] Reinforcement learning involves training a model to make decisions in dynamic environments by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms can include Q-learning and policy gradient methods.

[0246] Examples of specific machine learning algorithms that can be deployed include logistic regression, a type of supervised learning algorithm used for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example type of machine learning algorithm is Naive Bayes, a supervised learning algorithm used for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random forests are another type of supervised learning algorithm used for classification, regression, and other tasks. Random forests build an ensemble of decision trees and combine their outputs to make predictions. Further examples include neural networks, which consist of layers of interconnected nodes (or neurons) that process information and make predictions based on input data. Matrix factorization is another type of machine learning algorithm used for recommendation systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to reveal hidden patterns or relationships in the data. Support Vector Machines (SVMs) are a type of supervised learning algorithm used for classification, regression, and other tasks. SVMs find hyperplanes that separate different classes in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer models. The choice of algorithm can depend on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0247] The performance of a machine learning model can be evaluated on a separate test dataset that was not used during training, ensuring that the model can generalize to new, unseen data.

[0248] While this paper discusses several specific examples of machine learning algorithms, at least some of the principles discussed can be applied to other machine learning algorithms. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and transformers, as well as more traditional machine learning algorithms such as decision trees, random forests, and gradient boosting, can be used in a wide variety of machine learning applications.

[0249] Two typical types of problems in machine learning are classification problems and regression problems. Classification problems (also known as categorization problems) aim to classify an item into one of several category values ​​(e.g., is the object an apple or an orange?). Regression algorithms aim to quantify some items (e.g., by providing values ​​as real numbers).

[0250] Generating a trained machine learning program 1502 may include multiple stages forming part of a machine learning pipeline 1400, said multiple stages including, for example... Figure 14 The following stages are shown:

[0251] Data Collection and Preprocessing 1402: This stage may include acquiring and cleaning data to ensure it is suitable for use in machine learning models. This stage may also include removing duplicates, handling missing values, and converting the data to a suitable format.

[0252] Feature engineering 1404: This stage may include selecting and transforming training data 1506 to create features useful for predicting the target variable. Feature engineering may include (1) receiving features 1508 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying features 1508 in training data 1506 (e.g., unstructured or unlabeled data for unsupervised learning).

[0253] Model Selection and Training 1406: This stage may include selecting an appropriate machine learning algorithm and training it on preprocessed data. This stage may also involve splitting the data into training and test sets, using cross-validation to evaluate the model, and tuning hyperparameters to improve performance.

[0254] Model Evaluation 1408: This phase may include evaluating the performance of the trained model (e.g., a trained machine learning program 1502) on a separate test dataset. This phase can help determine whether the model is overfitting or underfitting, and whether the model is suitable for deployment.

[0255] Prediction 1410: This stage involves using a trained model (e.g., a trained machine learning program 1502) to generate predictions for new, unseen data.

[0256] Validation, refinement, or retraining 1412: This stage may include updating the model based on feedback generated from the prediction stage (such as new data or user feedback).

[0257] Deployment 1414: This phase may include integrating the trained model (e.g., a trained machine learning program 1502) into a wider system or application (e.g., a web service, mobile application, or Internet of Things (IoT) device). This phase may involve installing the API, building the user interface, and ensuring that the model is scalable and can handle large amounts of data.

[0258] Figure 15Further details of two example phases are shown: training phase 1504 (e.g., part of model selection and training 1406) and prediction phase 1510 (part of prediction 1410). Prior to training phase 1504, feature engineering 1404 is used to identify features 1508. This can include identifying informative, distinctive, and independent features for efficient operation of the trained machine learning program 1502 in pattern recognition, classification, and regression. In some examples, training data 1506 includes labeled data known for the pre-identified features 1508 and one or more outcomes. Each of the features 1508 can be a variable or attribute, such as various measurable properties of a process, item, system, or phenomenon represented by the dataset (e.g., training data 1506). By way of example only, features 1508 can also have different types such as numerical features, strings, and graphs, and can include one or more of content 1512, concepts 1514, attributes 1516, historical data 1518, and / or user data 1520.

[0259] In training phase 1504, the machine learning program can use training data 1506 to find correlations between features 1508 that influence prediction outcomes or prediction / inference data 1522. Using training data 1506 and the identified features 1508, the trained machine learning program 1502 is trained during training phase 1504, which is also during machine learning program training 1524. Machine learning program training 1524 evaluates feature 1508 values ​​when they are correlated with training data 1506. The result of the training is the trained machine learning program 1502 (e.g., a trained or learned model).

[0260] Furthermore, the training phase 1504 may involve machine learning, where the training data 1506 is structured (e.g., labeled during preprocessing). The trained machine learning program 1502 may implement a neural network 1526 capable of performing operations such as classification and clustering. In other examples, the training phase 1504 may involve deep learning, where the training data 1506 is unstructured, and the trained machine learning program 1502 implements a deep neural network 1526 capable of performing both feature extraction and classification / clustering operations.

[0261] In some examples, neural network 1526 may be generated during training phase 1504 and implemented within a trained machine learning program 1502. Neural network 1526 includes a hierarchical (e.g., layered) organization of neurons, where each layer consists of multiple neurons or nodes. Neurons in the input layer receive input data, while neurons in the output layer produce the network's final output. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0262] Each neuron in a neural network 1526 can operationally compute a function, such as an activation function, which takes as input a weighted sum of the outputs of neurons in the previous layer and a bias term. The output of this function is then passed as input to neurons in the next layer. If the output of the activation function exceeds a certain threshold, the output is passed from that neuron (e.g., a sending neuron) to a connecting neuron (e.g., a receiving neuron) in a subsequent layer. The connections between neurons have associated weights that define the influence of the input from the sending neuron to the receiving neuron. During the training phase, these weights are adjusted by a learning algorithm to optimize the network's performance. Different types of neural networks can use different activation functions and learning algorithms, affecting their performance on different tasks. The hierarchical organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs and generalize to new inputs not seen during training.

[0263] In some examples, neural network 1526 can also be one of several different types of neural networks, such as a single-layer feedforward network, a multilayer perceptron (MLP), an artificial neural network (ANN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a bidirectional neural network, a symmetric connection neural network, a deep belief network (DBN), a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder neural network (AE), a restricted Boltzmann machine (RBM), a Hopfield network, a self-organizing map (SOM), a radial basis function network (RBFN), a spiking neural network (SNN), a liquid state machine (LSM), an echo state network (ESN), a neural Turing machine (NTM), or a Transformer network, as examples only. Some machine learning models may include multiple neural networks 1526.

[0264] In addition to the training phase 1504, a validation phase can be performed on a separate dataset called the validation dataset. The validation dataset is used to tune the model's hyperparameters, such as the learning rate and regularization parameters. Tuning the hyperparameters improves the model's performance on the validation dataset.

[0265] Once the model is fully trained and validated, the testing phase allows it to be tested on a new dataset. The test dataset is used to evaluate the model's performance and ensure that it has not overfitted the training data.

[0266] In the prediction phase 1510, the trained machine learning program 1502 uses features 1508 to analyze the query data 1528 to generate inference, results, or predictions, as an example of prediction / inference data 1522. For example, during the prediction phase 1510, the trained machine learning program 1502 generates outputs. The query data 1528 is provided as input to the trained machine learning program 1502, and the trained machine learning program 1502 generates prediction / inference data 1522 as output in response to receiving the query data 1528.

[0267] In some examples, the trained machine learning program 1502 can be a generative artificial intelligence (AI) model. Generative AI is a term that can refer to any type of artificial intelligence that can create new content. For example, generative AI can produce text, images, videos, audio, code, or synthetic data. In some examples, the generated content may resemble the original data, but is not exactly the same.

[0268] Some of the techniques that can be used in generative AI are:

[0269] CNNs: CNNs can be used for image recognition and computer vision tasks. For example, a CNN can be designed to extract features from an image by using filters or kernels that scan the input image and highlight important patterns.

[0270] RNNs: For example, RNNs can be used to process sequential data such as speech, text, and time series data. RNNs employ feedback loops that allow them to capture temporal dependencies and remember past inputs.

[0271] GAN: A GAN can consist of two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can "fool" the discriminator network, while the discriminator network tries to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.

[0272] Variational Autoencoders (VAEs): VAEs encode input data into a latent space (e.g., a compressed representation) and then decode it back to output data. The latent space can be manipulated to generate new variations of the output data. VAEs can use self-attention mechanisms to process input data, allowing them to handle long text sequences and capture complex dependencies.

[0273] Transformer models: Transformer models use attention mechanisms to learn relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data such as text or speech, as well as non-sequential data such as images or code.

[0274] In the generative AI example, the prediction / inference data 1522 may include predictions, translations, summaries, answers, media content, or a combination thereof.

[0275] In some examples, once a model is trained (e.g., on a large training dataset), it can be fine-tuned on a smaller dataset more specific to the problem at hand. Therefore, even after deployment 1414, the machine learning pipeline 1400 can return to the model selection and training 1406 stage for fine-tuning. For example, a computer vision model can be fine-tuned on a dataset of medical images to create a model for medical image classification. Fine-tuning uses a pre-trained model as a starting point and trains it further on the new dataset, typically with a lower learning rate and fewer epochs. This allows the model to adapt its learned feature representations to the new dataset. The pre-trained weights are updated during fine-tuning to better fit the new data, rather than training a new model "from scratch." This transfer of knowledge from the pre-trained dataset can yield better performance compared to a model trained only on a smaller dataset. The machine learning pipeline 1400 can then proceed again to model evaluation 1408 to evaluate the fine-tuned model before deployment.

[0276] The examples described in this paper can utilize visual semantic machine learning models. In some examples, a visual semantic machine learning model consists of an image encoder, a text encoder, and a device for fusing information from both encoders. The image and text encoders can use a transformer architecture. As mentioned above, BLIP and BLIP-2 are non-limiting examples of visual semantic machine learning models with multimodal capabilities.

[0277] Data communication architecture

[0278] Figure 16This is a schematic diagram illustrating the structure of message 1600 according to some examples, which is generated by interactive client 104 for transmission to another interactive client 104 via interactive server 124. The content of a particular message 1600 can be used to populate a file stored in... Figure 1 The message table 304 is located within database 128 and is accessible by interactive server 124. Similarly, the content of message 1600 can be stored in memory as "in-transit" or "in-flight" data for user system 102 or interactive server 124. Message 1600 is shown as including the following example components:

[0279] Message Identifier 1602: A unique identifier that identifies message 1600.

[0280] Message text payload 1604: The text to be generated by the user via the user interface of user system 102 and included in message 1600.

[0281] Message image payload 1606: Image data captured by the camera device component of user system 102 or retrieved from the memory component of user system 102 and included in message 1600. The image data for the sent or received message 1600 can be stored in image table 314.

[0282] Message video payload 1608: Video data captured by the camera device component or retrieved from the memory component of the user system 102 and included in message 1600. The video data for the sent or received message 1600 can be stored in video table 312.

[0283] Message audio payload 1610: Audio data captured by the microphone or retrieved from the memory component of the user system 102 and included in message 1600.

[0284] Message enhancement data 1612: Represents enhancement data (e.g., filters, stickers, or other annotations or enhancements) to be applied to the message image payload 1606, message video payload 1608, or message audio payload 1610 of message 1600. Enhancement data for the sent or received message 1600 can be stored in enhancement table 310.

[0285] Message duration parameter 1614: A parameter value, in seconds, indicating the amount of time that the content of the message (e.g., message image payload 1606, message video payload 1608, message audio payload 1610) will be presented to the user via the interactive client 104 or made accessible to the user.

[0286] Message geolocation parameter 1616: Geolocation data (e.g., latitude and longitude coordinates) associated with the message's content payload. Multiple message geolocation parameter 1616 values ​​may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within the message image payload 1606 or a specific video within the message video payload 1608).

[0287] Message collection identifier 1618: An identifier value that identifies one or more content sets (e.g., "Stories" identified in set table 316) associated with a specific content item in the message image payload 1606 of message 1600. For example, the identifier value can be used to associate multiple images within the message image payload 1606 with multiple content sets, respectively.

[0288] Message Tag 1620: Each message 1600 can be labeled with multiple tags, each of which indicates the subject of the content included in the message payload. For example, in the case where a specific image depicts an animal (e.g., a lion) is included in the message image payload 1606, a tag value can be included within the message tag 1620 indicating the relevant animal. Tag values ​​can be manually generated based on user input, or can be automatically generated using, for example, image recognition.

[0289] Message sender identifier 1622: An identifier (e.g., message sending system identifier, email address, or device identifier) ​​indicating the user of the user system 102 on which message 1600 is generated and from which message 1600 is sent.

[0290] Message receiver identifier 1624: An identifier (e.g., message sending and receiving system identifier, email address, or device identifier) ​​indicating the user of the user system 102 to which message 1600 is addressed.

[0291] The content (e.g., values) of each component of message 1600 can be pointers to locations in tables where the content data values ​​are stored. For example, the image value in message image payload 1606 can be a pointer to a location within image table 314 (or the address of a location within image table 314). Similarly, the value in message video payload 1608 can point to data stored in video table 312, the value stored in message enhancement 1612 can point to data stored in enhancement table 310, the value stored in message collection identifier 1618 can point to data stored in collection table 316, and the values ​​stored in message sender identifier 1622 and message receiver identifier 1624 can point to user records stored in entity table 306.

[0292] Figure 17 A network environment 1700 is shown, according to some examples, in which a head-worn wearable device 1702 (e.g., a head-worn XR or AR device) can be implemented. Figure 17 A high-level functional block diagram of an example head-mounted wearable device 1702, communicatively coupled to a mobile user equipment 1738 and a server system 1732 via a suitable network 1740, is provided. Similar devices can be used... Figure 17 The head-mounted wearable devices 1702 or network of devices shown herein perform one or more of the techniques described herein.

[0293] The head-mounted wearable device 1702 includes at least one of a camera device, such as a visible light camera 1712, an infrared camera, and a transmitter 1714. The head-mounted wearable device 1702 includes other sensors 1716, such as motion sensors or eye-tracking sensors. A user equipment 1738 may be able to connect to the head-mounted wearable device 1702 using both communication links 1734 and 1736. The user equipment 1738 is connected to a server system 1732 via a network 1740. The network 1740 may include any combination of wired and wireless connections.

[0294] The head-worn device 1702 includes a display arrangement with several components. This arrangement includes two image displays 1704 with optical components. In other examples, the head-worn device 1702 may then include one image display or more than two image displays.

[0295] The two displays include one associated with the left side of the head-worn device 1702 and one associated with the right side of the head-worn device 1702. The head-worn device 1702 also includes an image display driver 1708, an image processor 1710, a low-power circuitry system 1726, and a high-speed circuitry system 1718. The image display 1704 is used to present images and videos to the user of the head-worn device 1702, including images that can provide a graphical user interface.

[0296] Image display driver 1708 commands and controls the image display of each image display 1704. Image display driver 1708 may deliver image data directly to each image display in image display 1704 for presentation, or may need to convert the image data into a signal or data format suitable for delivery to each image display device. For example, image data may be video data formatted according to a compression format (e.g., H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc.), and still image data may be formatted according to a compression format (e.g., Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tag Image File Format (TIFF), or Exchangeable Image File Format (Exif), etc.).

[0297] The head-wearable device 1702 may include a frame and a stem (or temple) extending from the side of the frame, or another component to facilitate the user wearing the head-wearable device 1702. Figure 17 The head-mounted wearable device 1702 also includes a user input device 1706 (e.g., a touch sensor or a press button), which includes an input surface on the head-mounted wearable device 1702. The user input device 1706 is configured to receive input selections from the user to manipulate a graphical user interface of the presented image.

[0298] Figure 17 The components shown for the head-worn wearable device 1702 are located on one or more circuit boards (e.g., printed circuit boards (PCBs) or flexible PCBs) in the frame or legs. Alternatively or additionally, the depicted components may be located in the blocks, frames, hinges, or nose bridge of the head-worn wearable device 1702. The left and right sides of the head-worn wearable device 1702 may each include digital imaging elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0299] The head-mounted wearable device 1702 includes a memory 1722 that stores instructions for performing a subset or all of the functions described herein. The memory 1722 may also include a storage device. For example... Figure 17 As further shown, the high-speed circuit system 1718 includes a high-speed processor 1720, a memory 1722, and a high-speed wireless circuit system 1724. Figure 17 In this embodiment, the image display driver 1708 is coupled to the high-speed circuit system 1718 and operated by the high-speed processor 1720 to drive the left and right image displays of the image display 1704. The high-speed processor 1720 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the head-worn device 1702. The high-speed processor 1720 includes the processing resources required to manage high-speed data transmission over a communication link 1736 to a wireless local area network (WLAN) using the high-speed wireless circuit system 1724. In some examples, the high-speed processor 1720 executes an operating system (e.g., a LINUX operating system or another such operating system for the head-worn device 1702), and the operating system is stored in memory 1722 for execution. Among other duties, the high-speed processor 1720, which executes the software architecture of the head-worn device 1702, manages data transmission with the high-speed wireless circuit system 1724. In some examples, the high-speed wireless circuit system 1724 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 1702.11 communication standard (also referred to herein as Wi-Fi). TM In other examples, other high-speed communication standards can be implemented using the high-speed wireless circuit system 1724.

[0300] The low-power wireless circuit system 1730 and high-speed wireless circuit system 1724 of the head-mounted wearable device 1702 may include a short-range transceiver (Bluetooth). TM ) and wireless wide area network transceivers, wireless local area network transceivers, or wide area network transceivers (e.g., cellular or Wi-Fi) TM User equipment 1738, including transceivers that communicate via communication links 1734 and 1736, can be implemented using the architectural details of head-mounted wearable device 1702, as can other components of network 1740.

[0301] Memory 1722 includes any storage device capable of storing various data and applications, including camera data generated by visible light imaging device 1712, sensor 1716, and image processor 1710, and images generated by image display driver 1708 for display on image display 1704, etc. While memory 1722 is shown as integrated with high-speed circuitry system 1718, in other examples, memory 1722 may be a separate, independent component of head-mounted wearable device 1702. In some such examples, circuit wiring may provide a connection from image processor 1710 or low-power processor 1728 to memory 1722 via a chip including high-speed processor 1720. In other examples, high-speed processor 1720 may manage addressing of memory 1722, such that low-power processor 1728 will activate high-speed processor 1720 whenever a read or write operation involving memory 1722 is required.

[0302] like Figure 17 As shown, the low-power processor 1728 or high-speed processor 1720 of the head-worn wearable device 1702 may be coupled to a camera device (visible light camera 1712, infrared camera, and transmitter 1714), an image display driver 1708, a user input device 1706 (e.g., a touch sensor or a press button), and a memory 1722. The head-worn wearable device 1702 also includes a sensor 1716, for example, as referenced below. Figure 18 The sensor 1716 may be a motion component 1830, a position component 1834, an environmental component 1832, and a biometric component 1828. Specifically, the head-worn device 1702 uses the motion component 1830 and the position component 1834 in conjunction with a video feed from one of the visible light camera devices 1712 to determine and track the position and orientation (“pose”) of the head-worn device 1702 relative to a reference frame or another object using techniques such as structure of motion (SfM) or VIO.

[0303] In some examples, and such as Figure 17 As shown, the head-mounted wearable device 1702 is connected to a host computer. For example, the head-mounted wearable device 1702 is paired with a user device 1738 via a communication link 1736, or connected to a server system 1732 via a network 1740. The server system 1732 may be one or more computing devices as part of a service or network computing system, for example, it includes a processor, memory, and a network communication interface to communicate with the user device 1738 and the head-mounted wearable device 1702 via the network 1740.

[0304] User equipment 1738 includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via network 1740, communication link 1734, or communication link 1736. User equipment 1738 may also store at least a portion of the instructions for implementing the functions described herein.

[0305] The output components of the head-worn wearable device 1702 include visual components, such as displays (e.g., one or more liquid crystal displays (LCDs)), one or more plasma display panels (PDPs), one or more light-emitting diode (LED) displays, one or more projectors, or one or more waveguides. The image display 1704 of the optical components is driven by an image display driver 1708. The output components of the head-worn wearable device 1702 also include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. The input components (e.g., user input device 1706) of the head-worn wearable device 1702, user equipment 1738, and server system 1732 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens providing position and force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0306] The head-mounted wearable device 1702 may optionally include additional peripheral device elements. Such peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the head-mounted wearable device 1702. For example, peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such components described herein.

[0307] For example, biometric components include those for detecting expressions (e.g., hand gestures, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion components include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Position components include positioning sensor components (e.g., GPS receiver components) for generating position coordinates, and Wi-Fi for generating positioning system coordinates. TMAlternatively, it could be a Bluetooth™ transceiver, an altitude sensor (e.g., an altimeter or barometer that detects air pressure and determines altitude), an orientation sensor (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from user equipment 1738 via communication link 1736 through a low-power wireless circuit system 1730 or a high-speed wireless circuit system 1724.

[0308] Machine architecture

[0309] Figure 18 This is a schematic representation of machine 1800, within which instructions 1802 (e.g., software, program, application, app, or other executable code) can be executed to cause machine 1800 to perform any or more of the methods discussed herein. For example, instructions 1802 can cause machine 1800 to perform any or more of the methods described herein. Instructions 1802 transform the general, unprogrammed machine 1800 into a specific machine 1800 programmed to perform the described and illustrated functions in the described manner. Machine 1800 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 1800 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1800 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), AR devices, XR devices, virtual reality (VR) devices, smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 1802 specifying actions to be taken by machine 1800. Furthermore, although only a single machine 1800 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 1802 to perform any or more of the methods discussed herein. For example, machine 1800 may include user system 102 or any of a plurality of server devices forming part of interactive server system 110. In some examples, machine 1800 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0310] Machine 1800 may include a processor 1804, a memory 1806, and an input / output (I / O) unit 1808 that can be configured to communicate with each other via a bus 1810. In the example, processor 1804 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 1812 and 1814 that execute instruction 1802. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 18 Multiple processors 1804 are shown, but machine 1800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0311] Memory 1806 includes main memory 1816, static memory 1818, and memory cell 1820, all of which are accessible by processor 1804 via bus 1810. Main memory 1806, static memory 1818, and memory cell 1820 store instructions 1802 that implement any one or more of the methods or functions described herein. Instructions 1802 may also reside wholly or partially in main memory 1816, static memory 1818, machine-readable medium 1822 within memory cell 1820, at least one processor of processor 1804 (e.g., within the processor's cache memory), or any suitable combination thereof during execution by machine 1800.

[0312] I / O component 1808 may include various components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurement results, etc. The specific I / O component 1808 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine is unlikely to include such a touch input device. It should be recognized that I / O component 1808 may include... Figure 18Many other components are not shown. In various examples, I / O component 1808 may include user output component 1824 and user input component 1826. User output component 1824 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input component 1826 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and force of a touch or touch gesture), audio input components (e.g., microphones), etc.

[0313] In other examples, I / O component 1808 may include biometric component 1828, motion component 1830, environmental component 1832, or position component 1834, as well as a wide range of other components. For example, biometric component 1828 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition).

[0314] Any biometric data collected by the biometric component is captured and stored only with user approval and is deleted upon user request. Furthermore, such biometric data may be used for very limited purposes (e.g., identification and verification). To ensure the restricted and authorized use of biometric information and other personally identifiable information (PII), access to this data is limited to authorized personnel (if access to the data occurs). Any use of biometric data may be strictly limited to identification and verification purposes, and such biometric data is not shared or sold to any third party without the user's explicit consent. In addition, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0315] The moving part 1830 includes an acceleration sensor part (e.g., an accelerometer), a gravity sensor part, and a rotation sensor part (e.g., a gyroscope).

[0316] Environmental component 1832 includes, for example, one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases for safety purposes or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0317] Regarding the camera device, user system 102 may have a camera device system including, for example, a front-facing camera on the front surface of user system 102 and a rear-facing camera on the rear surface of user system 102. The front-facing camera may be used, for example, to capture still images and videos (e.g., "selfies") of the user of user system 102, which can then be enhanced with the aforementioned enhancement data (e.g., filters). For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera device mode, which are similarly enhanced with the enhancement data. In addition to the front-facing and rear-facing cameras, user system 102 may also include a 360° camera for capturing 360° photos and videos.

[0318] Furthermore, the camera system of user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quadruple, or quintuple rear camera configurations on the front and rear sides of user system 102. For example, these multi-camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0319] The position component 1834 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and the like.

[0320] Various technologies can be used to achieve communication. I / O component 1808 also includes communication component 1836, which is operable to couple machine 1800 to network 1838 or device 1840 via a suitable coupling or connection. For example, communication component 1836 may include a network interface component or another suitable device that interfaces with network 1838. In further examples, communication component 1836 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, Bluetooth component, etc. TM Components (e.g., Bluetooth) TM Low-power, Wi-Fi components, and other communication components for providing communication via other modes. Device 1840 can be any peripheral device from another machine or various peripheral devices (e.g., a peripheral device coupled via USB).

[0321] Furthermore, the communication component 1836 can detect identifiers, or include components operable to detect identifiers. For example, the communication component 1836 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph™, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from the tag). Additionally, various information can be obtained via the communication component 1836, such as location obtained via Internet Protocol (IP) geolocation, location obtained via Wi-Fi® signal triangulation, location obtained by detecting NFC beacon signals that can indicate a specific location, etc.

[0322] Various memories (e.g., main memory 1816, static memory 1818, and the memory of processor 1804) and storage unit 1820 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1802) may, when executed by processor 1804, enable various operations to implement the disclosed examples.

[0323] Instructions 1802 can be sent or received over network 1838 via a transmission medium using a network interface device (e.g., a network interface component included in communication component 1836) and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 1802 can be sent or received via a transmission medium coupled to device 1840 (e.g., peer-to-peer coupling).

[0324] Software Architecture

[0325] Figure 19 This is a block diagram 1900 illustrating a software architecture 1902 that can be installed on any one or more of the devices described herein. The software architecture 1902 is supported by hardware such as a machine 1904 including a processor 1906, memory 1908, and I / O components 1910. In this example, the software architecture 1902 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 1902 includes layers such as an operating system 1912, libraries 1914, frameworks 1916, and applications 1918. Operationally, application 1918 activates API calls 1920 via the software stack and receives messages 1922 in response to API calls 1920.

[0326] Operating system 1912 manages hardware resources and provides public services. Operating system 1912 includes, for example, kernel 1924, services 1926, and drivers 1928. Kernel 1924 acts as an abstraction layer between the hardware layer and other software layers. For example, kernel 1924 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 1926 can provide other public services to other software layers. Drivers 1928 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 1928 may include display drivers, camera drivers, and bluetooth drivers. TM Or BLUETOOTH TM Low-power drives, flash drives, serial communication drives (e.g., USB drives), Wi-Fi drives, audio drives, power management drives, etc.

[0327] Library 1914 provides common low-level infrastructure used by application 1918. Library 1914 may include system library 1930 (e.g., the C standard library), which provides functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 1914 may include API library 1932, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codecs, Joint Picture Experts Group (JPEG or JPG), or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, which provides various relational database functions), web libraries (e.g., WebKit, which provides web browsing capabilities), etc. Library 1914 may also include various other libraries 1934 to provide many other APIs to application 1918.

[0328] Framework 1916 provides common high-level infrastructure for use by Application 1918. For example, Framework 1916 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 1916 can provide a wide range of other APIs that can be used by Application 1918, some of which may be specific to a particular operating system or platform.

[0329] In the example, application 1918 may include home application 1936, contact application 1938, browser application 1940, book reader application 1942, location application 1944, media application 1946, messaging application 1948, game application 1950, and a wide variety of other applications such as third-party application 1952. Application 1918 is a program that performs the functions defined in the program. One or more applications 1918 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 1952 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ SDK) may be mobile software running on a mobile operating system such as iOS™, Android™, Windows® Phone, or other mobile operating systems. In this example, third-party application 1952 may activate API call 1920 provided by operating system 1912 to facilitate the functions described herein.

[0330] Example

[0331] In view of the above-mentioned implementation methods, this application discloses the following list of examples, wherein a feature of a single example or more than one feature of an example is combined together, and optionally, is combined with one or more features of one or more other examples, which are also further examples falling within the disclosure of this application.

[0332] Example 1 is a system comprising: at least one processor; and at least one memory unit storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: accessing a first input image and a second input image, the second input image being generated by applying a target augmented reality (AR) effect to the first input image; providing the first input image and the second input image to a first visual semantic machine learning model to obtain an output describing at least one feature of the target AR effect, the first visual semantic machine learning model being fine-tuned by a second visual semantic machine learning model on training data including a plurality of training samples, each training sample including a first training image, a second training image, and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image; selecting a description of the target AR effect based on the output of the first visual semantic machine learning model; and storing the description of the target AR effect in association with an identifier of the target AR effect.

[0333] In Example 2, the subject of Example 1 includes, wherein fine-tuning of the second visual semantic machine learning model includes: using the training samples to adapt the transformer of the second visual semantic machine learning model to obtain the first visual semantic machine learning model.

[0334] In Example 3, the subject of Example 2 includes the following: the transformer fine-tunes the first visual semantic machine learning model between the image encoder and the large language model of the first visual semantic machine learning model by adapting the attention parameters of the transformer while keeping the image encoder and the large language model fixed.

[0335] In Example 4, the subject of any of Examples 2 to 3 includes, wherein the second visual semantic machine learning model is fine-tuned by low-rank adaptation (LoRA).

[0336] In Example 5, the subject of any one of Examples 1 to 4 includes, wherein the at least one feature comprises at least one visual feature, and the first visual semantic machine learning model is fine-tuned to describe the at least one visual feature based on the visual transition from the first input image to the second input image caused by the target AR effect.

[0337] In Example 6, the subject of any one of Examples 1 to 5 includes, wherein the first visual semantic machine learning model uses the first input image and the second input image to perform an AR effect-specific addition of explanatory text operation.

[0338] In Example 7, the subject of any one of Examples 1 to 6 includes the operation further comprising: applying the target AR effect to the first input image to obtain the second input image.

[0339] In Example 8, the subject of any one of Examples 1 to 7 includes providing the first input image and the second input image as a stitched image to the first visual semantic machine learning model.

[0340] In Example 9, the subject of Example 8 includes the operation further comprising: generating the stitched image by stitching the first input image and the second input image together.

[0341] In Example 10, the subject of Example 9 includes the following: the stitching of the first input image and the second input image includes: positioning the first input image and the second input image relative to each other in a predetermined spatial arrangement to obtain the stitched image.

[0342] In Example 11, the subject of any one of Examples 1 to 10 includes, wherein, for each training sample, the first training image and the second training image are stitched together to form a stitched training image.

[0343] In Example 12, the subject of any one of Examples 1 to 11 includes, wherein the first input image is a frame of a base video and the second input image is a frame of an enhanced video generated by rendering the target AR effect on the base video.

[0344] In Example 13, the subject of Example 12 includes the operation further comprising: identifying frames of the enhanced video as corresponding in time to frames of the base video; extracting frames of the base video to obtain the first input image; and extracting frames of the enhanced video to obtain the second input image.

[0345] In Example 14, the subject of any one of Examples 1 to 13 includes, wherein the description of the target AR effect is a natural language text description, and the storage of the description of the target AR effect includes: indexing the target AR effect based on the description of the target AR effect in the interactive system.

[0346] In Example 15, the subject of Example 14 includes the operation further comprising: receiving a search query from a user device of a user of the interactive system; matching the target AR effect with the search query based on an index of the target AR effect; detecting a user selection of the target AR effect; and, in response to the matching of the target AR effect with the search query and the user selection of the target AR effect, causing image data to be presented at the user device, the target AR effect being applied to the image data.

[0347] In Example 16, the subject of any one of Examples 1 to 15 includes the operation further comprising: determining the category of the target AR effect based on the description of the target AR effect; and storing the category of the target AR effect in association with the identifier of the target AR effect.

[0348] In Example 17, the subject of any one of Examples 1 to 16 includes, wherein the target AR effect is one of a plurality of AR effects supported by an interactive system, and the first visual semantic machine learning model is used to generate a description of each of the plurality of AR effects.

[0349] In Example 18, the subject of Example 17 includes the storage, wherein associating the description of the target AR effect with the identifier of the target AR effect, the storage includes storing the description of the target AR effect in a database that maps each of the plurality of AR effects to a corresponding description.

[0350] Example 19 is a method comprising: accessing a first input image and a second input image, the second input image being generated by applying a target augmented reality (AR) effect to the first input image; providing the first input image and the second input image to a first visual semantic machine learning model to obtain an output describing at least one feature of the target AR effect, the first visual semantic machine learning model being fine-tuned by a second visual semantic machine learning model on training data including a plurality of training samples, each training sample including a first training image, a second training image and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image; selecting a description of the target AR effect based on the output of the first visual semantic machine learning model; and storing the description of the target AR effect in association with an identifier of the target AR effect.

[0351] Example 20 is a non-transitory computer-readable storage medium storing one or more instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: accessing a first input image and a second input image, the second input image being generated by applying a target augmented reality (AR) effect to the first input image; providing the first input image and the second input image to a first visual semantic machine learning model to obtain an output describing at least one feature of the target AR effect, the first visual semantic machine learning model being fine-tuned by a second visual semantic machine learning model on training data including a plurality of training samples, each training sample including a first training image, a second training image, and a training description of a given AR effect, and the second training image being generated by applying the given AR effect to the first training image; selecting a description of the target AR effect based on the output of the first visual semantic machine learning model; and storing the description of the target AR effect in association with an identifier of the target AR effect.

[0352] Example 21 is at least one machine-readable medium including instructions that, when executed by a processing circuitry system, cause the processing circuitry system to perform operations for implementing any one of Examples 1 to 20.

[0353] Example 22 is a device that includes means for implementing any one of Examples 1 to 20.

[0354] Example 23 is a system for implementing any one of Examples 1 through 20.

[0355] Example 24 is a method for implementing any one of Examples 1 through 20.

[0356] in conclusion

[0357] As used in this disclosure, the term "machine learning model" (or simply "model") can refer to a single, independent model or a combination of models. The term can also refer to a system, component, or module that includes a machine learning model and one or more supporting or supplementary components that do not necessarily perform machine learning tasks.

[0358] As used in this disclosure, phrases such as "at least one of A, B, or C," "at least one of A, B, or C," and "at least one of A, B, and C" should be interpreted as selecting at least one from the group including "A, B, and C." In this disclosure, unless explicitly stated otherwise in conjunction with specific examples, this phrasing does not imply "at least one of A, at least one of B, and at least one of C." As used in this disclosure, the example "at least one of A, B, or C" will cover any of the following selections: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, and {A, B, C}.

[0359] Unless the context explicitly requires it, throughout the specification and claims, the words “comprising,” “including,” etc., shall be interpreted in an inclusive sense rather than an exclusive or exhaustive sense; that is, meaning “including but not limited to.” As used herein, the terms “connection,” “coupled,” or any variation thereof mean any direct or indirect connection or coupling between two or more elements; the coupling or connection between elements may be physical, logical, or a combination thereof. Furthermore, when used in this application, the words “in this document,” “above,” “below,” and words with similar meanings refer to the entire application and not any particular part of the application. Where the context permits, the use of singular or plural terms may also include the plural or singular, respectively. When referring to a list of two or more items, the word “or” covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list. Similarly, with respect to a list of two or more items, the word “and / or” covers all the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list.

[0360] The various features, steps, operations, and processes described herein can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain methods, processing boxes, or operations may be omitted in some implementations.

[0361] For ease of reference, the term "operation" is used to refer to an element in the accompanying drawings of this disclosure, and it will be understood that each "operation" may identify one or more operations, processes, actions or steps, and may be performed by one or more components.

[0362] While some examples (such as those depicted in the accompanying figures) include a specific order of operations, that order may be changed without departing from the scope of this disclosure. For example, some operations in the depicted operations may be performed in parallel or in a different order that does not substantially affect the functionality described in the examples. In other examples, different components of an example device or system implementing the example methods may perform their functions substantially simultaneously or in a specific order.

[0363] Glossary

[0364] For example, "carrier signal" refers to any intangible medium or other intangible medium capable of storing, encoding, or carrying instructions executed by a machine and including digital or analog communication signals. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0365] For example, "client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0366] For example, "communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a Common Old-Style Telephone Service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, coupling can enable any data transmission technology of various types, such as single-carrier radio transmission technology (1xRTT), evolved data optimization (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0367] For example, a “component” refers to a logical or physical entity having boundaries defined by functional or subroutine calls, branch points, APIs, or other technical definitions that partition or modularize a particular processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be an encapsulated functional hardware unit designed for use with other components, and part of a program that typically performs a related function. A component can constitute a software component (e.g., code implemented on a machine-readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuit systems temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine), which is uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be appreciated that a decision can be made, for cost and time considerations, whether to implement a hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to include tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider the example of a hardware component being temporarily configured (e.g., programmed), without needing to configure or instantiate each hardware component at any given time. For example, in cases where the hardware components include a general-purpose processor that is configured as a dedicated processor via software, this general-purpose processor can be configured at different times as its respective dedicated processor (e.g., including different hardware components). The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from the memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where a particular processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the execution of related operations in a “cloud computing” environment or as a “Software as a Service” (SaaS) operation. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some examples, the processor or processor-implemented component may reside in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component may be distributed across multiple geographic locations.

[0368] For example, "computer-readable storage medium" refers to both machine-readable storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and can be used interchangeably in this disclosure.

[0369] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0370] For example, a "non-transitory computer-readable storage medium" refers to a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0371] For example, "signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions executable by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.

[0372] "User equipment" means, for example, a device that is accessed, controlled, or owned by a user and that the user interacts with to perform actions or interactions (including interactions with other users or computer systems).

Claims

1. A system comprising: At least one processor; as well as At least one memory unit stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: Access a first input image and a second input image, the second input image being generated by applying a target augmented reality (AR) effect to the first input image; The first input image and the second input image are provided to a first visual semantic machine learning model to obtain an output describing at least one feature of the target AR effect. The first visual semantic machine learning model is obtained by fine-tuning a second visual semantic machine learning model on training data including multiple training samples. Each training sample includes a first training image, a second training image and a training description of a given AR effect. The second training image is generated by applying the given AR effect to the first training image. Based on the output of the first visual semantic machine learning model, a description of the target AR effect is selected; and The description of the target AR effect is stored in association with the identifier of the target AR effect.

2. The system according to claim 1, wherein, Fine-tuning the second visual semantic machine learning model includes: using the training samples to adapt the transformer of the second visual semantic machine learning model to obtain the first visual semantic machine learning model.

3. The system according to claim 2, wherein, The transformer fine-tunes the first visual semantic machine learning model by adapting the attention parameters of the transformer to the image encoder and the large language model while keeping the image encoder and the large language model fixed.

4. The system according to claim 2, wherein, The second visual semantic machine learning model is fine-tuned through low-rank adaptation (LoRA).

5. The system according to claim 1, wherein, The at least one feature includes at least one visual feature, and the first visual semantic machine learning model is fine-tuned to describe the at least one visual feature based on the visual transition from the first input image to the second input image caused by the target AR effect.

6. The system according to claim 1, wherein, The first visual semantic machine learning model uses the first input image and the second input image to perform AR effect-specific explanatory text addition operations.

7. The system according to claim 1, wherein the operation further comprises: The target AR effect is applied to the first input image to obtain the second input image.

8. The system according to claim 1, wherein, The first input image and the second input image are provided as a stitched image to the first visual semantic machine learning model.

9. The system according to claim 8, wherein the operation further comprises: The stitched image is generated by stitching together the first input image and the second input image.

10. The system according to claim 9, wherein, The stitching of the first input image and the second input image includes: positioning the first input image and the second input image relative to each other in a predetermined spatial arrangement to obtain the stitched image.

11. The system according to claim 1, wherein, For each training sample, the first training image and the second training image are stitched together to form a stitched training image.

12. The system according to claim 1, wherein, The first input image is a frame of a base video, and the second input image is a frame of an enhanced video generated by rendering the target AR effect on the base video.

13. The system according to claim 12, further comprising: The frames of the enhanced video are identified as corresponding to the frames of the base video in time; Extract frames from the base video to obtain the first input image; as well as Frames of the enhanced video are extracted to obtain the second input image.

14. The system according to claim 1, wherein, The description of the target AR effect is a natural language text description, and the storage of the description of the target AR effect includes: In the interactive system, the target AR effect is indexed based on the description of the target AR effect.

15. The system of claim 14, further comprising: Receive search queries from the user's user device in the interactive system; Based on the index of the target AR effect, the target AR effect is matched with the search query; User selection for detecting the target AR effect; as well as In response to the matching of the target AR effect with the search query and the user selection of the target AR effect, image data is presented on the user device, and the target AR effect is applied to the image data.

16. The system according to claim 1, wherein the operation further comprises: The category of the target AR effect is determined based on the description of the target AR effect; as well as The category of the target AR effect is stored in association with the identifier of the target AR effect.

17. The system according to claim 1, wherein, The target AR effect is one of a plurality of AR effects supported by the interactive system, and the first visual semantic machine learning model is used to generate a description of each of the plurality of AR effects.

18. The system according to claim 17, wherein, Storing the description of the target AR effect in association with the identifier of the target AR effect includes storing the description of the target AR effect in a database that maps each of the plurality of AR effects to a corresponding description.

19. A method comprising: Access a first input image and a second input image, the second input image being generated by applying a target augmented reality (AR) effect to the first input image; The first input image and the second input image are provided to a first visual semantic machine learning model to obtain an output describing at least one feature of the target AR effect. The first visual semantic machine learning model is obtained by fine-tuning a second visual semantic machine learning model on training data including multiple training samples. Each training sample includes a first training image, a second training image and a training description of a given AR effect. The second training image is generated by applying the given AR effect to the first training image. Based on the output of the first visual semantic machine learning model, a description of the target AR effect is selected; as well as The description of the target AR effect is stored in association with the identifier of the target AR effect.

20. A non-transitory computer-readable storage medium storing instructions, said instructions, when executed by at least one processor, causing said at least one processor to perform an operation, said operation comprising: Access a first input image and a second input image, the second input image being generated by applying a target augmented reality (AR) effect to the first input image; The first input image and the second input image are provided to a first visual semantic machine learning model to obtain an output describing at least one feature of the target AR effect. The first visual semantic machine learning model is obtained by fine-tuning a second visual semantic machine learning model on training data including multiple training samples. Each training sample includes a first training image, a second training image and a training description of a given AR effect. The second training image is generated by applying the given AR effect to the first training image. Based on the output of the first visual semantic machine learning model, a description of the target AR effect is selected; as well as The description of the target AR effect is stored in association with the identifier of the target AR effect.

Images