Generating text from images using machine learning

Abstract

Implementations described herein relate to methods, devices, and computer-readable media to generate text. In some implementations, a method includes providing a prompt to a large language model (LLM). The prompt may include a task description and respective metadata associated with one or more images. The method further includes providing, to the LLM user-specific data. The method further includes generating, by the LLM, text responsive to the prompt based on the task description, the metadata for the one or more images, and the user-specific data. The text is personalized based on the data of the user account and is responsive to the task description. The method further includes causing the text to be displayed in a user interface.

Description

GENERATING TEXT FROM IMAGES USING MACHINE LEARNINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 519,735, entitled “GENERATING TEXT FROM IMAGES USING MACHINE LEARNING,” filed on August 15, 2023, the content of which is incorporated herein in its entirety.BACKGROUND

[0002] Users may use smartphones and cameras to capture images and videos. Images and videos captured by a user may be stored in an image library of the user. The user may organize images into albums or collections, may edit images, may share images with other users, etc. The user may enter text descriptive of the images and / or events associated with the images. Manual entry of text may be cumbersome and inefficient.

[0003] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.SUMMARY

[0004] Various implementations relate to a computer-implemented method that includes providing a prompt to a large language model (LLM), wherein the prompt includes a task description and respective metadata associated with one or more images. The one or more images are from an image library associated with a user account. The method further includes providing, to the LLM, data of the user account, wherein the data includes userspecific data including one or more of: person names, place names, activity types, object names, and dates. The method further includes, generating, by the LLM, text responsive to the prompt based on the task description, the metadata for the one or more images, and the data of the user account. The text is personalized based on the data of the user account and is responsive to the task description. The method further includes causing the text to be displayed in a user interface.

[0005] In some implementations, the method further includes generating respective captions for the one or more images using an image captioning model and providing the captions to the LLM. In these implementations, the generating the text by the LLM is further based on the respective captions.

[0006] In some implementations, the one or more images are in an image album. In these implementations, the method further includes receiving a user request for an album title for the image album, and in response to the user request, selecting the task description that specifies that the text is for use as the album title. The text includes one or more suggested album titles.

[0007] In some implementations, the method further includes receiving a search query from a user and identifying the one or more images from the image library associated with the user account that match the search query. In these implementations, the user interface includes at least one of the one or more images.

[0008] In some implementations, the method further includes receiving user input text and providing the user input text to the LLM, wherein the generating the text by the LLM is based at least in part on the user input text.

[0009] In some implementations, the method further includes, after causing the text and at least one of the one or more images to be displayed in the user interface, receiving user input text and providing the user input text to the LLM. The method further includes generating, by the LLM, revised text based at least in part on the user input text and causing the revised text to be displayed in the user interface. In some implementations, the method further includes automatically updating the data of the user account based on the user input text.

[0010] In some implementations, the method further includes accessing a knowledge repository based on the metadata associated with the one or more images, to obtain additional information, and wherein generating the text is further based on the additional information.

[0011] In some implementations, the user interface includes at least one of the one or more images. In some implementations, at least a portion of the text is overlaid on the at least one of the one or more images.

[0012] Some implementations include a computing device comprising a processor and a memory’ coupled to the processor, with instructions stored thereon. The instructions, executed by the processor, cause the processor to perform operations comprising providing a prompt to a large language model (LLM), wherein the prompt includes a task description and respective metadata associated with one or more images, wherein the one or more images are from an image library associated with a user account. The operations further include providing, to the LLM, data of the user account, wherein the data includes userspecific data including one or more of person names, place names, activity types, object names, and dates. The operations further include generating, by the LLM, text responsive to the prompt based on the task description, the metadata for the one or more images, and the data of the user account, wherein the text is personalized based on the data of the user account and is responsive to the task description, The operations further include causing the text to be displayed in a user interface.

[0013] In some implementations, the operations further include generating respective captions for the one or more images using an image captioning model and providing the captions to the LLM, wherein the generating the text by the LLM is further based on the respective captions.

[0014] In some implementations, the one or more images are in an image album, and the operations further include receiving a user request for an album title for the image album and in response to the user request, selecting the task description that specifies that the text is for use as the album title, wherein the text includes one or more suggested album titles. In some of these implementations, the operations further include receiving a search query from a user and identifying the one or more images from the image library associated with the user account that match the search query, wherein the user interface includes at least one of the one or more images.

[0015] In some implementations, the operations further include receiving user input text and providing the user input text to the LLM, wherein the generating the text by the LLM is based at least in part on the user input text.

[0016] In some implementations, the operations further include, after causing the text and at least one of the one or more images to be displayed in the user interface, receiving user input text, providing the user input text to the LLM, generating, by the LLM, revised text based at least in part on the user input text, and causing the revised text to be displayed in the user interface.

[0017] In some implementations, the operations further include accessing a knowledge repository' based on the metadata associated with the one or more images, to obtain additional information, and wherein generating the text is further based on the additional information.

[0018] Some implementations include a non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations that include providing a prompt to a large language model (LLM), wherein the prompt includes a task description and respective metadata associated with one or more images, wherein the one or more images are from an image library associated with a user account. The operations further include providing, to the LLM, data of the user account, wherein the data includes user-specific data including one or more of: person names, place names, activity types, object names, and dates. The operations further include generating, by the LLM, text responsive to the prompt based on the task description, the metadata for the one or more images, and the data of the user account, wherein the text is personalized based on the data of the user account and is responsive to the task description. The operations further include causing the text to be displayed in a user interface.

[0019] In some implementations, the instructions cause the processor to perform further operations that include generating respective captions for the one or more images using an image captioning model and providing the captions to the LLM, wherein the generating the text by the LLM is further based on the respective captions.

[0020] In some implementations, the instructions cause the processor to perform further operations that include receiving a search query from a user and identifying the one or more images from the image library associated with the user account that match the search query, wherein the user interface includes at least one of the one or more images.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Fig. 1 is a block diagram of an example network environment which may be used for one or more implementations described herein.

[0022] Fig. 2 is a block diagram illustrating an example method, according to some implementations.

[0023] Fig. 3 is an example system architecture, according to some implementations.

[0024] Fig. 4 illustrates example images.

[0025] Figs. 5A and 5B illustrate example user interfaces, according to some implementations.

[0026] Fig. 6 is a block diagram of an example device which may be used for one or more implementations described herein.DETAILED DESCRIPTION

[0027] Implementations described herein relate to use of machine learning techniques to generate text based on images and / or image metadata. In some implementations, a large language model (LLM) may generate text in response to a prompt that includes a task description (e.g., “generate an album title for these images,’7“generate a summary description for these images,” etc.) and metadata of a set of images. The LLM may also be provided user-specific data, user input text, and / or image captions as input, additional to the prompt.

[0028] The described implementations leverage knowledge about images and users associated with images to automatically generate text for various tasks. For example, the task may be generating a poem descriptive of a set of images (e.g., “my summer in a poem”);generating a description of a birthday party' (e.g., “Jake turned 30. His best friends Ryan, Anil, and Ghazala attended. The black forest cake was fabulous. Ryan played the piano. Jake got a surprise gift - a guitar!’’) for use in a visual story of the birthday party with multiple images; creating an actionable element from an image (e.g., “This is a receipt from Store X for Item Y. The total spend was $10.24, including taxes of $1.24. Click here to add to your expense report’’); etc.

[0029] The described implementations may automatically generate text for a given set of images based on information about an image and the user associated with the image. The described implementations reduce the computational resources required to render user interfaces that a user can use to enter text.

[0030] FIG. 1 illustrates a block diagram of an example network environment 100, which may be used in some implementations described herein. In some implementations, network environment 100 includes one or more server systems, e.g., server system 102 in the example of FIG. 1. Server system 102 can communicate with a network 130, for example. Serv er system 102 can include a server device 104 and a database 106 or other storage device. In some implementations, server device 104 may provide image management application 156b and one or more machine learning models 158b. In FIG. 1 and the remaining figures, a letter after a reference number, e.g., “156a,” represents a reference to the element having that particular reference number. A reference number in the text without a following letter, e.g., “156,” represents a general reference to embodiments of the element bearing that reference number.

[0031] Network environment 100 also can include one or more client devices, e.g., client devices 120, 122, 124, and 126, which may communicate with each other and / or with server system 102 via network 130. Network 130 can be any type of communication network, including one or more of the Internet, local area networks (LAN), wireless networks, switch or hub connections, etc. In some implementations, network 130 can include peer-to-peer communication between devices, e.g., using peer-to-peer wireless protocols (e.g., Bluetooth®, Wi-Fi Direct, etc.), etc. One example of peer-to-peer communications between two client devices 120 and 122 is shown by arrow 132.

[0032] For ease of illustration, FIG. 1 shows one block for server system 102, server device 104, database 106, and four blocks for client devices 120, 122, 124, and 126. Blocks102, 104, and 106 may represent multiple systems, server devices, and network databases, and the blocks can be provided in different configurations than shown. For example, server system 102 can represent multiple server systems that can communicate with other server systems via the network 130. In some implementations, server system 102 can include cloud hosting servers, for example. In some examples, database 106 and / or other storage devices can be provided in server system block(s) that are separate from server device 104 and can communicate with server device 104 and other server systems via network 130.

[0033] Also, there may be any number of client devices. Each client device can be any type of electronic device, e.g., desktop computer, laptop computer, portable or mobile device, cell phone, smartphone, tablet computer, television, TV set top box or entertainment device, wearable devices (e g., display glasses or goggles, wristwatch, headset, armband, jewelry, etc.), personal digital assistant (PDA), media player, game device, etc. Some client devices may also have a local database similar to database 106 or other storage. In some implementations, network environment 100 may not have all of the components shown and / or may have other elements including other types of elements instead of, or in addition to, those described herein.

[0034] In various implementations, end-users Ul, U2, U3, and U4 may communicate with server system 102 and / or each other using respective client devices 120, 122. 124, and 126. In some examples, users Ul, U2, U3, and U4 may interact with each other via applications running on respective client devices and / or server system 102 and / or via a network sen-ice, e.g., a social network service or other type of network service, implemented on server system 102. For example, respective client devices 120, 122. 124, and 126 may communicate data to and from one or more server systems, e.g., server system 102.

[0035] In some implementations, the server system 102 may provide appropriate data to the client devices such that each client device can receive communicated content or shared content uploaded to the server system 102 and / or a netw ork service. In some examples, users U1-U4 can interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0036] A network service implemented by server system 102 can include a system allowing users to perform a variety of communications, form links and associations, upload and post shared content such as images (e.g., individual images, image collections such asimage albums, stories where a series of images are shown in succession, image creations such as animations, etc.), text, video, audio, and other ty pes of content, and / or perform other functions. For example, a client device can display received data such as content posts sent or streamed to the client device and originating from a different client device via a server and / or network senice (or from the different client device directly), or originating from a server system and / or network service. In some implementations, client devices can communicate directly with each other, e.g.. using peer-to-peer communications between client devices as described above. In some implementations, a “user’ can include one or more programs or virtual entities, as well as persons that interface with the system or network.

[0037] In some implementations, any of client devices 120, 122, 124, and / or 126 can provide one or more applications. For example, as shown in FIG. 1, client device 120 may provide image management application 156a. Client device 120 may also include one or more machine learning models 158a. Client devices 122-126 may also provide similar applications.

[0038] Image management application 156a may be implemented using hardware and / or software of client device 120. In different implementations, image management application 156a may be a standalone client application, e.g., executed on any of client devices 120-126, or may work in conjunction with image management application 156b provided on server system 102. Image management application 156a may provide various functions related to images and / or videos. For example, such functions may include one or more of capturing images using a camera; programmatically analyzing images, e.g., to assign labels that are indicative of subject matter of the images; modifying images; storing images in an image library or database, providing user interfaces to view images; generating, viewing, and sharing image-based creations (e.g., stories where a series of images are shown in succession, animations, social media posts, etc.) or image collections such as image albums, etc.

[0039] In some implementations, image management application 156 may enable a user to manage the library or database that stores images. For example, a user may use a backup functionality of image application 156a on a client device (e.g.. any of client devices 120- 126) to back up local images or videos on the client device to a server device, e.g., server device 104. For example, the user may manually select one or more images or videos to bebacked up, or specify backup setings that identify images or videos to be backed up. Backing up an image or video to a server device may include transmiting the image or video to the server for storage by the server, e.g., in coordination with image application 156b on server device 104.

[0040] In some implementations, client device 120 and / or client devices 122-126 may include one or more machine learning models 158. A client machine learning model 158a may be implemented using hardware and / or software of client device 120. In various implementations, machine learning model 158a may be usable directly on any of client devices 120-126, or may work in conjunction with machine learning model 158b provided on server system 102.

[0041] Machine learning models 158 may provide various functions related to digital maps. For example, such functions may include automatically generating labels for the image (e.g.. based on recognizing one or more objects or entities in the image); generating a caption for the image (e.g., an image captioning model may be trained to generate short text that is suitable for use as a caption, based on the image, image metadata, and / or labels produced by another model or other techniques). In some implementations, a language model (e.g., a large language model) may be included in machine learning models 158.

[0042] In some implementations, the language model may be a machine learning model that is capable of generating responses to text prompts provided as input to the model. In some implementations, the text prompts may include a task description for a task and image metadata, and the language model may generate output text. In some implementations, the language model may provide an application programming interface (API) for other applications, e.g., to enable the other application such as image management application 156 to utilize the language model to generate text by providing a prompt. Language model 158 may utilize data, e.g.. images, image metadata including image labels, data from a user account (e.g., of a user associated with a client device 120), etc. The data may be stored locally on client device 120, and / or may retrieve information from server device 104. In some implementations, the language model may be a multimodal model, e.g., that can take as input non-text data such as images, videos, binary files, or other types of data.

[0043] In different implementations, client device 120 and / or server device 104 may provide other applications (not shown) that may be applications that provide various typesof functionality, e.g., calendar, address book, e-mail, web browser, shopping, transportation (e.g., taxi, train, airline reservations, etc.), entertainment (e.g., a music player, a video player, a gaming application, etc.), social networking (e.g., messaging or chat, audio / video calling, sharing images / video, etc.) and so on. In some implementations, one or more of other applications may be standalone applications that execute on client device 120. In some implementations, one or more of other applications may access a server system, e.g., server system 102, that provides data and / or functionality of other applications. In various implementations, data associated with the one or more other applications may be stored in a user account. With user permission, the data of the user account may be provided to machine learning models 158.

[0044] A user interface on a client device 120, 122, 124, and / or 126 can enable the display of user content and other content, including images, image albums, video, data, and other content as well as communications, privacy settings, notifications, and other data. Such a user interface can be displayed using software on the client device, software on the server device, and / or a combination of client software and server software executing on server device 104, e.g., application software or client software in communication with server system 102. The user interface can be displayed by a display device of a client device or server device, e.g., a touchscreen or other display screen, projector, etc. In some implementations, application programs running on a server system can communicate with a client device to receive user input at the client device and to output data such as visual data, audio data, etc. at the client device.

[0045] Other implementations of features described herein can use any type of system and / or service. For example, other networked services (e g., connected to the Internet) can be used instead of or in addition to a social networking service. Any ty pe of electronic device can make use of features described herein. Some implementations can provide one or more features described herein on one or more client or server devices disconnected from or intermittently connected to computer networks. In some examples, a client device including or connected to a display device can display content posts stored on storage devices local to the client device, e.g., received previously over communication networks.

[0046] An image as referred to herein can include a digital image having pixels with one or more pixel values (e.g., color values, brightness values, etc ). An image can be a still image (e.g., still photos, images with a single frame, etc.), a dynamic image (e.g.,animations, motion images, animated GIFs, cinemagraphs where a portion of the image includes motion while other portions are static, etc.), or a video (e.g., a sequence of images or image frames that may include audio). While the remainder of this document refers to an image as a static image, it may be understood that the techniques described herein are applicable for dynamic images, video, etc. For example, implementations described herein can be used with still images (e.g., a photograph, or other image), videos, or dynamic images.

[0047] Fig. 2 is a flow diagram illustrating an example method 200 to generate text, according to some implementations. In some implementations, method 200 can be implemented, for example, on a server system 102 as shown in Fig. 1. In some implementations, some or all of the method 200 can be implemented on one or more client devices 120, 122, 124, or 126 as shown in Fig. 1, one or more server devices, and / or on both server device(s) and client device(s). In described examples, the implementing system includes one or more digital processors or processing circuitry ("processors"), and one or more storage devices (e.g., a database 106 or other storage). In some implementations, different components of one or more servers and / or clients can perform different blocks or other parts of the method 200. In some examples, a first device is described as performing blocks of method 200. Some implementations can have one or more blocks of method 200 performed by one or more other devices (e.g.. other client devices or server devices) that can send results or data to the first device.

[0048] In some implementations, the method 200, or portions of the method, can be initiated automatically by a system. In some implementations, the implementing system is a first device. For example, the method (or portions thereof) can be periodically performed, or performed based on one or more particular events or conditions, e.g., a user request, an image album being created, new images and / or videos being added to an image library associated with a user account, a client device entering an idle state, a predetermined time period having expired since the last performance of method 200, and / or one or more other conditions occurring which can be specified in settings read by the method.

[0049] Method 200 may begin at block 202. In block 202, it is checked whether user consent (e.g., user permission) has been obtained to use user data in the implementation of method 200. For example, user data can include images or videos stored on a client device (e.g., any of client devices 120-126), videos stored or accessed by a user, e.g., using a clientdevice, image metadata, image library associated with a user account, user data related to the use of an image management application, user preferences, etc. One or more blocks of the methods described herein may use such user data in some implementations.

[0050] If user consent has been obtained from the relevant users for which user data may be used in the method 200, then in block 204, it is determined that the blocks of the methods herein can be implemented with possible use of user data as descnbed for those blocks, and the method continues to block 212. If user consent has not been obtained, it is determined in block 206 that blocks are to be implemented without the use of user data, and the method continues to block 212. In some implementations, if user consent has not been obtained, blocks are implemented without the use of user data and with synthetic data and / or generic or publicly-accessible and publicly-usable data. In some implementations, if user consent has not been obtained, remaining blocks of method 200 are not performed.

[0051] In block 212, a prompt is provided to a large language model (LLM). The prompt may include a task description and respective image metadata associated with one or more images.

[0052] In some implementations, the prompt may be selected from a prompt library. Each prompt in the prompt library may include a task description. Some examples of such prompts include "generate an album title for images that have the following metadata,” “ generate a short phrase that summarizes the main attributes of images that have the following metadata.” “write a title based on the data that follows; include emojis,” “Catchy phrase based on the data below,” “suggest a phrase of ten or less words for the following images,” “image data for images captured in a trip follows, generate a ten-line description of the trip.” “These pictures were taken at a wedding. Write a short narrative describing the bride and groom, the location, the decor, and food,” and so on.

[0053] The prompt library may include a set of prompts that have been tested for use with the LLM. For example, such testing may be performed using sets of known images, with known attributes, and optionally, human-written text based on the sets of known images. The human-written text may be written by a human as a response to a particular task description. Various test prompts may be provided to the LLM and response text may be obtained from the LLM. The response text may be rated (e.g., assigned a score) based on the known attributes and / or the human- ritten text. Test prompts that result in responsesthat are high rated (e.g., meet a score threshold) for the sets of known images may be included in the set of prompts, while other tests prompts are excluded.

[0054] In some implementations, automatic evaluation may be performed based on a set of known images, based on characteristics of the generated titles, and machine learning models run on the titles to select prompts that are included in the prompt library. In some implementations, A / B testing may be utilized determine the prompts to include in the library, e g., prompts that cause the LLM to output response text to be selected by a user (e.g., as a caption, as an album title, etc. overresponse text generated by the LLM in response to other prompts, may be included in the prompt library.

[0055] In some implementations, the task description in the prompt may specify that text be generated by the LLM and a context for use of the generated text. For example, the context of use may be: an image caption (for a single image), respective image captions for two or more images, an album title for an album that includes one or more images, a snippet descriptive of a set of images, a search summary for images that match a search query, etc. In some implementations, the prompt may specify attributes for the text, e.g., length of text, language of text, style (e.g., funny, rhyming, formal, etc.), type (e.g., haiku, short poem), etc.

[0056] In some implementations, the prompt provided to the LLM for a task may be customized (modified) based on the image metadata (e.g., for images to be provided to the LLM with the prompt). If the images include certain metadata, the prompt for a task may be modified than when the images do not contain such metadata. The presence / absence of certain metadata, as well as the value of the metadata, may be utilized in modifying the prompt.

[0057] The prompt may also include respective image metadata associated with one or more images. The one or more images may be from an image library associated with a user account. For example, the image library may include images captured by a user and automatically added to the image library, images shared with the user by other users, images manually added to the library by the user, etc. A user's image library- may include different types (e.g., still images, motion images, animations, videos, screenshots, images of documents, etc.). The image metadata may include a plurality of attributes and associatedvalues. The attributes and the associated values may be stored as image metadata in the image.

[0058] For example, at the time of image capture, a camera type and capture settings (e.g., aperture, focal distance, etc.), date of capture, location of capture, etc. may be stored as image metadata. If the image is modified, the metadata may capture attributes such as modification time / date. modification tool (e.g.. image editing application), etc.

[0059] Further, in some implementations, an image may be programmatically analy zed to determine various attributes for the image. For example, with user permission, face detection and / or facial recognition techniques may be performed to identify faces in the image. If the user permits, names associated with the faces (e.g., as specified by the user for the image or from prior images that depict the face) may be stored as metadata.

[0060] In some implementations, the metadata may also include a label attribute with a plurality of values. The values in the label attribute may indicate various aspects of the image such as a type of environment (indoor / outdoor, sunny, cloudy, rainy, etc.), objects within the image (e.g., tree, sky, dog, etc ), image type (e.g., photograph, animation, video, screenshot, receipt, document, etc.), or any other value. The labels may be determined based on various image processing techniques such as object recognition, image segmentation, face detection, facial recognition, etc. In some implementations, the attributes may have associated confidence scores, where higher confidence scores are associated with higher likelihood of the attribute value being accurate. An example of image metadata is: {Faces: Jake, Cali; Location: 44.38867, -121.2155; Date: 2021-07-04; Labels: Dog, Animal, Pet. Child, Nature, Cloud, Summer, Tree, Sky, Eye, Plant, Sand; Camera: f / 2.0 1 / 360 2.22mm ISO35}.

[0061] Block 212 may be followed by block 214.

[0062] In block 214, data of the user account is provided to the LLM. In some implementations, the data may include user-specific data such as one or more of: person names, place names, activity types, object names, and dates. For example, the user-specific data may be stored as part of the user account. Person names may include the user’s name, names of other users known to the user (e.g., spouse, children, parents, relatives, friends, colleagues, etc.). In some implementations, person names may include names of pets orother animals. In some implementations, the person names may be provided by the user, may be retrieved (with permission) from a user’s contact list, etc.

[0063] In some implementations, place names in the data of the user account may include places of relevance to the user. For example, place names may include information descriptive of a user’s home (e.g., home address, city, state, country, etc.), workplace, homes of persons known to the user, places that a user has visited (e.g., restaurants, transit stops, stadiums, auditoriums, museums, tourist sites, outdoor locations, etc.). In some implementations, activity types may include activities that the user engages in or is interested in, e.g.. running, skating, cycling, skiing, hiking, dancing, singing, etc. In some implementations, object names may include objects that a user owns or likes, e.g., model of car, bicycle, or other vehicle; musical instruments; books; etc.

[0064] In some implementations, dates in the data of the user account may include dates of importance to the user, e.g., birthdates (self, spouse, children, parents, relatives, friends, colleagues, etc.); anniversaries (e.g., wedding anniversary, work anniversary, etc.); event dates (e.g., concerts, trips, sports events, etc. that the user participated in); etc.

[0065] In various implementations, data of the user account may’ include data provided by the user and / or if the user permits, data determined automatically from user activity'. For example, the user may provide their home address to a digital map application. Automatic determination of data from the user activity may be performed with specific user permission using any suitable analysis technique, e.g., natural language processing, text parsing, semantic analysis, text summarization, media summarization, etc. For example, if a user’s queries to a digital map include a place name, the place name may be included in the data of the user account. In another example, if the user saves a reminder about a flight, the flight date and origin and / or destination may be included in the data of the user account.

[0066] User information is accessed and processed with specific user permission, and according to settings provided by the user and in compliance with regulations. The user is given options to specify what data may be stored, the duration of storage, the use of such data, etc. The data of the user account is provided to the LLM for the specific purpose of generating text. Block 214 may be followed by block 216.

[0067] In block 216, captions may be generated on the one or more images and provided to the LLM. For example, the one or more images (and associated metadata) may beprovided to an image captioning model, e.g., a machine learned model that can generate captions (e.g.. short strings of text) based on image input. Each caption may correspond to a particular image of the one or more images and may be indicative of the subject matter of the image. In some implementations, block 21 may not be performed. Block 216 may be followed by block 218.

[0068] In block 218, user input text may be received and provided to the LLM. For example, the user input text may be received via a keyboard, as voice input, or via selection of a user interface element. For example, the user may provide text that provides additional information about the one or more images (e.g., that may not be present in the image metadata and / or the data of the user account). Such additional information may include, e.g., identifiers of people present in the images (e.g., ‘'the bearded man in the green jacket is Ryan,’’ “the horse I am riding is Daisy,”), objects in the images (e.g., “the boat is Ryan’s boat named Voyager,”), or other information (e.g., “these photos are from Ryan and Ghazala’s wedding,” "‘these are receipts from my trip to London,” etc.). In some implementations, with user permission, the data of the user account is updated based on the user input text. For example, the name “Ryan” may be associated with the “bearded man in the green jacket” in the user’s images. Block 214 may be followed by block 220.

[0069] In block 220, text is generated by the LLM. The text is responsive to the prompt and is based on the task description, the metadata for the one or more images, and the data of the user account. The text is personalized based on the data of the user account and is responsive to the task description. If captions are provided to the LLM (by execution of block 216) or if user input text is provided to the LLM (by execution of block 218), the captions and / or the user input text is also provided to the LLM.

[0070] For example, for the image that includes the boy named Jake and the dog named Cali, the data from the user account may indicate that the user has several pictures of Jake and Cali (e.g., multiple images in the user’s image library that have the labels “Jake” and “Cali”). Further, the image metadata (e.g., location coordinated) may indicate that the image was captured in Terrebonne, Oregon. The user’s image library may have no prior images of Jake and Cali in Oregon, indicating that this is likely their first time in Oregon. The age of the boy named Jake may be specified in the data of the user account. The image date (4 July 2021) indicates that this is likely a vacation (also since the user’s home may be indicated in the data of the user account as being in a different state). The presence of sand indicates thatthe boy and the dog are playing on the sand. The image metadata and the data from the user account provide contextual information to the LLM, while the task description provides indications of the type of output desired.

[0071] For example, in response to a task description in the prompt that indicates that the text is to be a caption of less than ten words for a single image, the LLM may generate personalized text such as "Jake and Cali building sandcastles” or ‘‘Jake and Cali playing in the sand.” In some implementations, the generated text may include emojis, e.g.,(dog), © (child), mi (castle), etc., e.g., “Cali © helped Jake build a mi .” On the other hand, if the task description is to generate a short (e.g., two-three sentence) description of the image, the LLM may generate personalized text such as “Summer 2021 was a fun vacation for Jake and Cali. Here, they are seen building a sandcastle.”

[0072] In implementations where block 216 is implemented and image captions are provided to the LLM, the generated text may further be based on the image captions. For example, if the one or more images include images with the captions “Rodeo time in Terrebonne,” “Jake’s first rodeo,” Cozy desert cabin,” “horse riding with Ryan and family,” “Sunset at Smith Rock State Park,” “hiking up the mountain,” “barbecue with Ryan,” etc. and the task description is to generate a title for an album that includes the one or more images, the album title may be “Outdoor fun in Oregon - roasting and riding” or “Terrebonne adventures with Ryan and family.”

[0073] In implementations where block 218 is implemented and user input text is provided to the LLM, the generated text may further be based on the user input text. For example, the generated text may utilize user input text, e.g., “riding the waves on the Voyager with Ryan” or “Ryan and Ghazala look stunning.”

[0074] In block 222, the text (generated by the LLM in block 220) is caused to be displayed in a user interface. For example, the text may be displayed as a summary’ of the one or more images, as an album title that includes the one or more images, as a title for a particular image, overlaid on top of a particular image, etc. Block 214 may be followed by block 222.

[0075] Various blocks of method 200 may be combined, split into multiple blocks, or be performed in parallel. For example, blocks 212 and 214 may be performed in parallel. Inanother example, blocks 216 and 218 may be performed in parallel. In some implementations, blocks 216 and / or 218 are not performed. Method 200, or portions thereof, may be repeated any number of times using additional inputs. For example, method 200 may be repeated for different prompts.

[0076] In some implementations, the one or more images for which the image metadata is provided to the LLM may be in an image album. In these implementations, a user request may be received for an album title for the image album. In response to the user request, a task description may be selected that specifies that the text (to be generated by the LLM) is for use as the album title and the task description is included in the prompt. One or more other parameters, e.g., minimum or maximum length (in words or characters), tone or style (e.g., whimsical, formal, funny, etc ), language, etc. for the text to be generated may be specified. In these implementations, the generated text may include a plurality of options - suggested album titles generated by the LLM - that are displayed to the user in a user interface. The user may select a particular title from the options as the album or modify the title and assign it to the album. The user may also choose to regenerate titles. In response to the user request, method 200 may be repeated to generate new text.

[0077] In some implementations, the method may further include receiving a search query from the user, e.g., via text input, voice input, image input, etc. For example, the user may specify a search “Oregon” (images that are associated with Oregon), “baseball” (images that are associated with baseball), “my birthday” (images that are associated with the user’s birthday), “mountains” etc. In image input, the user may provide a query image as the search.

[0078] In these implementations, the one or more images (for which metadata is provided to the LLM) may be identified as images from the image library associated with the user account that match the search query. In these examples, the text generated by the LLM may be descriptive of the search results. For example, in response to the search query “Oregon,” the identified one or more images may include three distinct sets of images from three different trips that the user took to Oregon. The generated text may include text descriptive of each trip, e.g.. “Portland food truck trail,” “Waterfalls,” and “Outdoors in Terrebonne.” The user interface may include the text and associated images from the one or more images, e.g., in a section associated with a respective trip. In this manner, the generated text may be a snippet that provides a summary7of matching results, organized into sections.

[0079] In another example, in response to the text query "mountains." the generated text may include an essay about various mountains that are featured in the image library’ of the user, along with images in the user’s image library’ for the mountains. In another example, in response to the text query ‘'pumpkin carving,” the generated text may include a plurality of sentences, each with an associated image of the one or more images, and the user interface may display the images in sequential order (based on image timestamp) with individual sentences, such that the sequential display provides a step-by-step view of the activity of pumpkin carving. In various implementations, the task description in the prompt may include a use context for the generated text (e.g., essay, section snippets, step-by-step view, etc.) and the generated text may be based on the use context.

[0080] In some implementations, after generating the text and causing the text to be displayed in a user interface, method 200 may further include receiving user input text. For example, the user may provide additional information that may not be included in the image metadata and / or the user account data or if present, may not be reflected in the generated text. In these implementations, the LLM may generate revised text based at least part on the user input text. For example, if the initial generated text is “dog with Jake,” the user may provide additional information as “the dog in the pictures is Cali” and the revised text may be “Cali with Jake.” In another example, the user input text may include a command, e.g., “I want text with emoji.” In the revised text, an emoji may be included. In these implementations, the revised text may be displayed in the user interface. The user may provide input text and request revisions any number of times.

[0081] In some implementations, method 200 may further include, with user permission, updating the data of the user account based on the user input text. For example, the update may include “the user has a dog named Cali” or “the user lives in California.” Updating the data in this manner may enable text generated by the LLM in future executions of method 200 to automatically take into the updated information and avoid errors.

[0082] In some implementations, prior to generating the text, the method may further include accessing a knowledge repository’ based on the metadata associated with the one or more images to obtain additional information. The knowledge repository may include a search index (e.g., of an Internet search engine), a digital map (e.g., including places and associated data such as location, place type, etc ), an entity database (e.g., that includes entity identifiers and associated data such as entity’ characteristics, e.g., names of dog breedsand corresponding characteristics, a product catalog (e.g.. with data about various products), etc. For example, if the metadata include a location (e.g., latitude / longitude) associated with an image, additional information may be fetched from a digital map that indicates entities associated with the location (e.g., restaurant X, library Y, bookstore Z, etc.). In these implementations, the additional information may be provided to the LLM (e.g., the LLM may implement an API or code to access the knowledge repository, or the additional information may be provided in the prompt). In these implementations, generating the text is further based on the additional information. For example, if the location coordinates are associated with an Italian restaurant named “Roberto’s Pizzeria,” the generated text may include the restaurant name. For example, the generated text for a set of food images all associated with the location may be “in food paradise at Roberto’s.”

[0083] Fig. 3 illustrates an example system architecture 300, according to some implementations. System architecture 300 includes an image library 302, a large language model 310, an image captioning model 308, user account data 312, and a user interface module 330.

[0084] Image library 302 may store a plurality7of images 304 associated with a user account, e.g., of a user of any of client devices 120-126 that uses image management application 156. In various implementations, image library 302 may be stored on server system 102 (e.g., in database 106), on a client device 120-126, or on a combination of server system 102 and client device 120, in accordance with user settings and permissions.

[0085] In some implementations, image library 302 may store image metadata in association with each image. Image metadata 306 may include metadata captured with the image, e.g., location, timestamp, camera settings, camera type, image format (e.g., raw, JPEG, etc.) In some implementations, image metadata may include labels generated based on programmatically analyzing images using object recognition, text recognition, entity recognition, image quality assessment, and / or other techniques. In some implementations, image metadata may include labels manually assigned by a user, e.g., “Jake,” “Cali,” etc.)

[0086] In operation, metadata 320 (of one or more images) from the image library may be provided to LLM 310. Further, in some implementations, images 316 (optionally with metadata 320) may7be provided to image captioning model 308 (e.g., a trained machinelearning model that generates captions for input images) to generate image captions 318.The image captions 318 may be provided to LLM 310.

[0087] In some implementations, user-specific data 316 from user account 312 may be provided to LLM 310. For example, user-specific data 316 may include person names, place names, activity types, object names, dates, or other information from user account 312. User-specific data 316 may be stored locally on a client device (120-126) or on a server system (e.g., in database 106).

[0088] Prompt 314 is provided to LLM 310. Prompt 314 may include a task description. In some implementations, user interface module 330 may provide a user interface that includes options for users to select one or more operations, and prompt 314 may be selected based on the user selections. For example, the user may select “suggest album title’' with reference to a plurality of images (e.g., displayed in the user interface) from the options. In another example, the user may select “search” and enter a search term, which may be used to select prompt 314. In some implementations, a prompt library’ (not shown) may be provided, and prompts may be selected from the prompt library. In some implementations, image metadata 320 may be provided together with or as a part of prompt 314.

[0089] In some implementations, user interface module 330 may enable a user to provide user input text 324. In these implementations, user input text 324 may be provided to LLM 310.

[0090] LLM 310 may generate text 322 based on provided inputs (e.g., prompt 314, image metadata 320, user-specific data 316, image captions 318, user input text 324, etc.). Generated text 322 may be provided to user interface module 330 which may display it in a user interface. For example, if the user selects “suggest album title,” the generated text may be shown in the user interface as a suggested album title. In another example, if the user selects “search,” the generated text may be displayed in the search interface as a summary or snippet descriptive of the search results (which may include images from image library’ 302 that match the search query ).

[0091] In some implementations, a user may provide user input text 324 after viewing generated text 322. In these implementations, LLM 310 may generate additional text 322 based on the user input text (while other inputs remain the same). Text generation and revision by LLM 310 based on user input text may be performed any number of times.

[0092] In various implementations, LLM 310 (or portions thereof) and / or image captioning model (308) may be implemented on a client device (120-126), on a server device 104, or a combination of client and server devices.

[0093] Fig. 4 illustrates example images from an image library'. As can be seen, image 402-404 each is an image with specific individuals, and other entities depicted. For example, image 402 depicts a child and a dog, image 404 depicts a child and a man, image 406 depicts a group of people riding horses, and image 408 depicts a group of people having a meal outdoors. In the example of Fig. 4, image 402 may be associated with the following image metadata: {Faces: Jake, Cali; Location: 44.38867, -121.2155; Date: 2021-07-04; Labels: Dog, Animal. Pet, Child. Nature, Cloud, Summer, Tree, Sky, Eye. Plant, Sand; Camera: f / 2.0 1 / 360 2.22mm ISO35}. The image metadata identifies the child as '‘Jake” and the dog as “Cali”. The image metadata further indicates a location of the image (latitude and longitude), a date the image was captured, etc. The image metadata also includes details about the camera and camera settings used to capture the image. The image metadata also includes labels associated with the image, e.g., “child,” “dog,” “sand,” etc.

[0094] Fig. 5A illustrates an example user interface 500. User interface 500 includes image 402. User interface 500 further includes text 502 that is generated, e g., using method 200 and by an LLM 310. In the example illustrated in Fig. 5 A, the generated text is “Call helped Jake build a castle” (where emojis associated with dog and castle are included in the text). As can be seen, when image metadata associated with image 402 (“dog” “child”), user-specific data (e.g., “my dog is Cali,” “my son's name is Jake”), and a prompt (e.g., “generate short phrase with emoji”) are provided as input to the LLM. the generated text is descriptive of the image content, including attributes that are not present in the image (e.g., names of individuals) are included in the generated text. Further, the activity7being depicted (“building a castle”) is included in the generated text even though the input data does not include the name of the activity.

[0095] Fig. 5B illustrates another example user interface 510. User interface 510 includes a plurality of images 512 arranged in a picture pile. User interface 510 further includes three text boxes 514, 516, and 518 that each include text that is generated e.g., using method 200 and by an LLM 310 (based on images 512, associated metadata, userspecific data, prompt, etc.). In the example illustrated in Fig. 5B, three options (514-518) are presented, and the user can select any of the options. The text in each box is a suggestedalbum title while the picture pile indicates contents of the album (which may include any number of images beyond the four images depicted in Fig. 5B). Alternatively, the user may enter their own album title in text box 520. The selected title is stored in association with the album.

[0096] Fig. 6 is a block diagram of an example device 600 which may be used to implement one or more features described herein. In one example, device 600 may be used to implement a client device, e.g., any of client devices 115 shown in Fig. 1. Alternatively, device 600 can implement a server device, e.g., server 101. In some implementations, device 600 may be used to implement a client device, a server device, or both client and server devices. Device 600 can be any suitable computer system, server, or other electronic or hardware device as described above.

[0097] One or more methods described herein can be run in a standalone program that can be executed on any type of computing device, a program run on a web browser, a mobile application (“app”) run on a mobile computing device (e.g., cell phone, smart phone, tablet computer, wearable device (wristwatch, armbandjewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head mounted display, etc.), laptop computer, etc.). In one example, a client / server architecture can be used, e.g., a mobile computing device (as a client device) sends user input data to a server device and receives from the server the final output data for output (e.g., for display). In another example, all computations can be performed within the mobile app (and / or other apps) on the mobile computing device. In another example, computations can be split between the mobile computing device and one or more server devices.

[0098] In some implementations, device 600 includes a processor 602, a memory 604, and input / output (I / O) interface 606. Processor 602 can be one or more processors and / or processing circuits to execute program code and control basic operations of the device 600. A “processor’ includes any suitable hardware system, mechanism or component that processes data, signals or other information. A processor may include a system with a general-purpose central processing unit (CPU) with one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA). an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for achieving functionality, a special-purpose processor toimplement neural network model-based processing, neural circuits, processors optimized for matrix computations (e.g.. matrix multiplication), or other systems. In some implementations, processor 602 may include one or more co-processors that implement neural-network processing. In some implementations, processor 602 may be a processor that processes data to produce probabilistic output, e.g., the output produced by processor 602 may be imprecise or may be accurate within a range from an expected output. Processing need not be limited to a particular geographic location or have temporal limitations. For example, a processor may perform its functions in “real-time,” “offline,” in a “batch mode,” etc. Portions of processing may be performed at different times and at different locations, by different (or the same) processing systems. A computer may be any processor in communication with a memory.

[0099] Memory' 604 is typically provided in device 600 for access by the processor 602, and may be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), Electrical Erasable Read-only Memory (EEPROM), Flash memory, etc., suitable for storing instructions for execution by the processor, and located separate from processor 602 and / or integrated therewith. Memory 604 can store software operating on the server device 600 by the processor 602, including an operating system 608, machine-learning application 630, other applications 612, and application data 614. Other applications 612 may include applications such as a data display engine, web hosting engine, image display engine, notification engine, social networking engine, etc. In some implementations, the machine-learning application 630 and other applications 612 can each include instructions that enable processor 602 to perform functions described herein, e.g., some or all of the method of Fig. 2.

[0100] Other applications 612 can include, e.g., image editing applications, media display applications, communication applications, web hosting engines or applications, mapping applications, media sharing applications, etc. One or more methods disclosed herein can operate in several environments and platforms, e.g., as a stand-alone computer program that can run on any type of computing device, as a web application having web pages, as a mobile application ("app") run on a mobile computing device, etc.

[0101] In various implementations, machine-learning application 634 may utilize Bayesian classifiers, support vector machines, neural networks, or other learning techniques. In some implementations, machine-learning application 630 may include atrained model 634, an inference engine 636, and data 632. In some implementations, data 632 may include training data, e.g.. data used to generate trained model 634. For example, training data may include any type of data such as text, images, audio, video, etc.

[0102] In some implementations, trained model 634 may be a large language model. The large language model may include a large number of parameters (e.g., thousands, millions, or billions of parameters). The large language model may be trained to respond to prompts with natural language text. The large language model may be trained based on a large corpus of text. In some implementations, the corpus of text used to train the large language model may include image descriptions, stories with images, image titles, emojis associated with images, social media posts including text and images, etc.

[0103] Training data may be obtained from any source, e.g., a data repository specifically marked for training, data for which permission is provided for use as training data for machine-learning, etc. In implementations where one or more users permit use of their respective user data to train a machine-learning model, e.g., trained model 634, training data may include such user data. In implementations where users permit use of their respective user data, data 632 may include permitted data such as images (e.g., photos or other user-generated images).

[0104] In some implementations, training data may include synthetic data generated for the purpose of training, such as data that is not based on user input or activity' in the context that is being trained, e.g., data generated from simulated photographs or other computergenerated images. In some implementations, machine-learning application 630 excludes data 632. For example, in these implementations, the trained model 634 may be generated, e.g., on a different device, and be provided as part of machine-learning application 630. In various implementations, the trained model 634 may be provided as a data file that includes a model structure or form, and associated weights. Inference engine 636 may read the data file for trained model 634 and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in trained model 634.

[0105] In some implementations, the trained model 634 may include one or more model forms or structures. For example, model forms or structures can include any type of neural- network, such as a linear network, a deep neural network that implements a plurality of layers (e.g., “hidden layers” between an input layer and an output layer, with each layerbeing a linear network), a convolutional neural network (e.g., a network that splits or partitions input data into multiple parts or tiles, processes each tile separately using one or more neural-network layers, and aggregates the results from the processing of each tile), a sequence-to-sequence neural network (e.g., a network that takes as input sequential data, such as words in a sentence, frames in a video, etc. and produces as output a result sequence), etc. The model form or structure may specify connectivity between various nodes and organization of nodes into layers.

[0106] For example, the nodes of a first layer (e.g., input layer) may receive data as input data 632 or application data 614. For example, when trained model 634 is a large language model, the input data may include a prompt (e.g., textual prompt). Subsequent intermediate layers may receive as input output of nodes of a previous layer per the connectivity7specified in the model form or structure. These layers may also be referred to as hidden layers or latent layers.

[0107] A final layer (e.g., output layer) produces an output of the machine-learning application. For example, the output may be generated text. In some implementations, model form or structure also specifies a number and / or type of nodes in each layer.

[0108] In different implementations, trained model 634 can include a plurality of nodes, arranged into layers per the model structure or form. In some implementations, the nodes may be computational nodes with no memory', e.g., configured to process one unit of input to produce one unit of output. Computation performed by a node may include, for example, multiplying each of a plurality of node inputs by a weight, obtaining a weighted sum, and adjusting the weighted sum with a bias or intercept value to produce the node output. In some implementations, the computation performed by a node may also include applying a step / activation function to the adjusted weighted sum. In some implementations, the step / activation function may be a nonlinear function. In various implementations, such computation may include operations such as matrix multiplication. In some implementations, computations by the plurality' of nodes may be performed in parallel, e.g., using multiple processors cores of a multicore processor, using individual processing units of a GPU, or special-purpose neural circuitry. In some implementations, nodes may include memory, e.g.. may be able to store and use one or more earlier inputs in processing a subsequent input. For example, nodes with memory may include long short-term memory (LSTM) nodes. LSTM nodes may use the memory' to maintain “state” that permits the nodeto act like a finite state machine (FSM). Models with such nodes may be useful in processing sequential data, e.g., words in a sentence or a paragraph, frames in a video, speech or other audio, etc.

[0109] In some implementations, trained model 634 may include embeddings or weights for individual nodes. For example, a model may be initiated as a plurality of nodes organized into layers as specified by the model form or structure. At initialization, a respective weight may be applied to a connection between each pair of nodes that are connected per the model form, e.g., nodes in successive layers of the neural network. For example, the respective weights may be randomly assigned, or initialized to default values. The model may then be trained, e.g., using data 632, to produce a result.

[0110] For example, training may include applying supervised learning techniques. In supervised learning, the training data can include a plurality of inputs (e.g., a set of grayscale images) and a corresponding expected output for each input (e.g., a set of groundtruth images corresponding to the grayscale images or other color images). Based on a comparison of the output of the model with the expected output, values of the weights are automatically adjusted, e.g., in a manner that increases a probability that the model produces the expected output when provided similar input.[OHl] In some implementations, training may include applying unsupervised learning techniques. In unsupervised learning, only input data may be provided and the model may be trained to differentiate data, e g., to cluster input data into a plurality of groups, where each group includes input data that are similar in some manner.

[0112] In some implementations, unsupervised learning may be used to produce knowledge representations, e.g., that may be used by machine-learning application 630. For example, unsupervised learning may be used to produce embeddings that are utilized by machine-learning application 630. In various implementations, a trained model includes a set of weights, or embeddings, corresponding to the model structure. In implementations where data 632 is omitted, machine-learning application 630 may include trained model 634 that is based on prior training, e.g., by a developer of the machine-learning application 630, by a third-party, etc. In some implementations, trained model 634 may include a set of weights that are fixed, e.g., downloaded from a server that provides the weights.

[0113] Machine-learning application 630 also includes an inference engine 636. Inference engine 636 is configured to apply the trained model 634 to data, such as application data 614, to provide an inference. In some implementations, inference engine 636 may include software code to be executed by processor 602. In some implementations, inference engine 636 may specify circuit configuration (e.g., for a programmable processor, for a field programmable gate array (FPGA). etc.) enabling processor 602 to apply the trained model. In some implementations, inference engine 636 may include software instructions, hardware instructions, or a combination. In some implementations, inference engine 636 may offer an application programming interface (API) that can be used by operating system 608 and / or other applications 612 to invoke inference engine 636, e.g., to apply trained model 634 to application data 614 to generate an inference. For example, the inference for a LLM model may be generated text.

[0114] Machine-learning application 630 may provide several technical advantages. For example, when trained model 634 is generated based on unsupervised learning, trained model 634 can be applied by inference engine 636 to produce knowledge representations (e.g., numeric representations) from input data, e.g., application data 614. For example, a model trained for image analysis may produce representations of images that have a smaller data size (e g., 1 KB) than input images (e.g., 10 MB). In some implementations, such representations may be helpful to reduce processing cost (e.g., computational cost, memory usage, etc.) to generate an output (e.g., a label, a classification, generated text in response to a prompt including image metadata and task description, etc.).

[0115] In some implementations, such representations may be provided as input to a different machine-learning application that produces output from the output of inference engine 636. In some implementations, knowledge representations generated by machinelearning application 630 may be provided to a different device that conducts further processing, e.g., over a network. In such implementations, providing the knowledge representations rather than the images may provide a technical benefit, e.g., enable faster data transmission with reduced cost. In another example, a model trained for clustering documents may produce document clusters from input documents. The document clusters may be suitable for further processing (e.g., determining whether a document is related to a topic, determining a classification category for the document, etc.) without the need to access the original document, and therefore, save computational cost.

[0116] In some implementations, machine-learning application 630 may be implemented in an offline manner. In these implementations, trained model 634 may be generated in a first stage and provided as part of machine-learning application 630. In some implementations, machine-learning application 630 may be implemented in an online manner. For example, in such implementations, an application that invokes machinelearning application 630 (e.g., operating system 608, one or more of other applications 612) may utilize an inference produced by machine-learning application 630, e.g., provide the inference to a user, and may generate system logs (e.g., if permitted by the user, an action taken by the user based on the inference; or if utilized as input for further processing, a result of the further processing). System logs may be produced periodically, e.g., hourly, monthly, quarterly, etc. and may be used, with user permission, to update trained model 634, e.g., to update embeddings for trained model 634.

[0117] In some implementations, machine-learning application 630 may be implemented in a manner that can adapt to particular configuration of device 600 on which the machine-learning application 630 is executed. For example, machine-learning application 630 may determine a computational graph that utilizes available computational resources, e.g., processor 602. For example, if machine-learning application 630 is implemented as a distributed application on multiple devices, machine-learning application 630 may determine computations to be carried out on individual devices in a manner that optimizes computation. In another example, machine-learning application 630 may determine that processor 602 includes a GPU with a particular number of GPU cores (e.g., 1000) and implement the inference engine accordingly (e.g., as 1000 individual processes or threads).

[0118] In some implementations, machine-learning application 630 may implement an ensemble of trained models. For example, trained model 634 may include a plurality of trained models that are each applicable to same input data. In these implementations, machine-learning application 630 may choose a particular trained model, e.g., based on available computational resources, success rate with prior inferences, etc. In some implementations, machine-learning application 630 may execute inference engine 636 such that a plurality of trained models is applied. In these implementations, machine-learning application 630 may combine outputs from applying individual models, e.g., using a votingtechnique that scores individual outputs from applying each trained model, or by choosingone or more particular outputs. Further, in these implementations, machine-learning application may apply a time threshold for applying individual trained models (e.g., 0.5 ms) and utilize only those individual outputs that are available within the time threshold. Outputs that are not received within the time threshold may not be utilized, e.g., discarded. For example, such approaches may be suitable when there is a time limit specified while invoking the machine-learning application, e.g., by operating system 608 or one or more applications 612.

[0119] In different implementations, machine-learning application 630 can produce different types of outputs. For example, machine-learning application 630 can provide representations or clusters (e.g.. numeric representations of input data), labels (e.g.. for input data that includes images, documents, etc ), phrases or sentences (e.g., descriptive of an image or video, suitable for use as a response to an input sentence, etc.), images (e.g., colorized or otherwise stylized images generated by the machine-learning application in response to input images, e.g., grayscale images), audio or video (e.g.. in response an input video, machine-learning application 630 may produce an output video with a particular effect applied, e.g., rendered in a comic-book or particular artist’s style, when trained model 634 is trained using training data from the comic book or particular artist, etc. In some implementations, machine-learning application 630 may produce an output based on a format specified by an invoking application, e.g., operating system 608 or one or more applications 612. In some implementations, an invoking application may be another machine-learning application. For example, such configurations may be used in generative adversarial networks, where an invoking machine-learning application is trained using output from machine-learning application 630 and vice-versa.

[0120] Any of software in memory 604 can alternatively be stored on any other suitable storage location or computer-readable medium. In addition, memory 604 (and / or other connected storage device(s)) can store one or more messages, one or more taxonomies, electronic encyclopedia, dictionaries, thesauruses, knowledge bases, message data, grammars, user preferences, and / or other instructions and data used in the features described herein. Memory' 604 and any other type of storage (magnetic disk, optical disk, magnetic tape, or other tangible media) can be considered "storage" or "storage devices."

[0121] I / O interface 606 can provide functions to enable interfacing the server device 600 with other systems and devices. Interfaced devices can be included as part of the device600 or can be separate and communicate with the device 600. For example, network communication devices, storage devices (e.g., memory and / or database 106). and input / output devices can communicate via I / O interface 606. In some implementations, the I / O interface can connect to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, camera, scanner, sensors, etc.) and / or output devices (display devices, speaker devices, printers, motors, etc ).

[0122] Some examples of interfaced devices that can connect to I / O interface 606 can include one or more display devices 620 that can be used to display content, e.g., images, video, and / or a user interface of an output application as described herein. Display device 620 can be connected to device 600 via local connections (e.g.. display bus) and / or via networked connections and can be any suitable display device. Display device 620 can include any suitable display device such as an LCD, LED, or plasma display screen, CRT, television, monitor, touchscreen, 3-D display screen, or other visual display device. For example, display device 620 can be a flat display screen provided on a mobile device, multiple display screens provided in a goggles or headset device, or a monitor screen for a computer device.

[0123] The I / O interface 606 can interface to other input and output devices. Some examples include one or more cameras which can capture images. Some implementations can provide a microphone for capturing sound (e.g., as a part of captured images, voice commands, etc.), audio speaker devices for outputting sound, or other input and output devices.

[0124] For ease of illustration. Fig. 6 shows one block for each of processor 602, memory7604, I / O interface 606, and software blocks 608, 612, and 630. These blocks may represent one or more processors or processing circuitries, operating systems, memories, I / O interfaces, applications, and / or software modules. In other implementations, device 600 may not have all of the components shown and / or may have other elements including other types of elements instead of, or in addition to, those show n herein. While some components are described as performing blocks and operations as described in some implementations herein, any suitable component or combination of components of environment 100. device 600. similar systems, or any suitable processor or processors associated with such a system, may perform the blocks and operations described.

[0125] Methods described herein can be implemented by computer program instructions or code, which can be executed on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product including a non-transitory computer readable medium (e.g., storage medium), such as a magnetic, optical, electromagnetic, or semiconductor storage medium, including semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), flash memory, a rigid magnetic disk, an optical disk, a solid-state memory drive, etc. The program instructions can also be contained in, and provided as, an electronic signal, for example in the form of software as a sendee (SaaS) delivered from a server (e.g., a distributed system and / or a cloud computing system). Alternatively, one or more methods can be implemented in hardware (logic gates, etc.), or in a combination of hardware and software. Example hardware can be programmable processors (e.g., Field-Programmable Gate Array (FPGA), Complex Programmable Logic Device), general purpose processors, graphics processors, Application Specific Integrated Circuits (ASICs), and the like. One or more methods can be performed as part of or component of an application running on the system, or as an application or software running in conjunction with other applications and operating system.

[0126] Although the description has been described with respect to particular implementations thereof, these particular implementations are merely illustrative, and not restrictive. Concepts illustrated in the examples may be applied to other examples and implementations.

[0127] Further to the descriptions above, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs, or features described herein may enable collection of user information (e.g., information about a user's images, a user's address book, social network, social actions, or activities, profession, a user’s preferences, or a user’s location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot bedetermined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0128] Note that the functional blocks, operations, features, methods, devices, and systems described in the present disclosure may be integrated or divided into different combinations of systems, devices, and functional blocks as would be known to those skilled in the art. Any suitable programming language and programming techniques may be used to implement the routines of particular implementations. Different programming techniques may be employed, e.g., procedural or object-oriented. The routines may execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, the order may be changed in different particular implementations. In some implementations, multiple steps or operations shown as sequential in this specification may be performed at the same time.

Claims

CLAIMS1. A computer-implemented method, the method comprising: providing a prompt to a large language model (LLM), wherein the prompt includes: a task description; and respective metadata associated with one or more images, wherein the one or more images are from an image library associated with a user account; providing, to the LLM, data of the user account, wherein the data includes user-specific data including one or more of: person names, place names, activity types, obj ect names, and dates; generating, by the LLM, text responsive to the prompt based on the task description, the metadata for the one or more images, and the data of the user account, wherein the text is personalized based on the data of the user account and is responsive to the task description: and causing the text to be displayed in a user interface.

2. The computer-implemented method of claim 1, further comprising: generating respective captions for the one or more images using an image captioning model; and providing the captions to the LLM. wherein the generating the text by the LLM is further based on the respective captions.

3. The computer-implemented method of claim 1, wherein the one or more images are in an image album, and the method further comprises: receiving a user request for an album title for the image album; and in response to the user request, selecting the task description that specifies that the text is for use as the album title, wherein the text includes one or more suggested album titles.

4. The computer-implemented method of claim 1, further comprising: receiving a search queiy from a user; and identifying the one or more images from the image library associated with the user account that match the search quei ', wherein the user interface includes at least one of the one or more images.

5. The computer-implemented method of claim 1, further comprising: receiving user input text; and providing the user input text to the LLM, wherein the generating the text by the LLM is based at least in part on the user input text.

6. The computer-implemented method of claim 1, further comprising, after causing the text and at least one of the one or more images to be displayed in the user interface: receiving user input text; providing the user input text to the LLM; generating, by the LLM, revised text based at least in part on the user input text; and causing the revised text to be displayed in the user interface.

7. The computer-implemented method of claim 6, further comprising, automatically updating the data of the user account based on the user input text.

8. The computer-implemented method of claim 1, further comprising, accessing a knowledge repositor}^ based on the metadata associated with the one or more images, to obtain additional information, and wherein generating the text is further based on the additional information.

9. The computer-implemented method of claim 1, wherein the user interface includes at least one of the one or more images.

10. The computer-implemented method of claim 9, wherein at least a portion of the text is overlaid on the at least one of the one or more images.

11. A computing device comprising: a processor; and a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising: providing a prompt to a large language model (LLM), wherein the prompt includes: a task description; andrespective metadata associated with one or more images, wherein the one or more images are from an image library associated with a user account; providing, to the LLM, data of the user account, wherein the data includes user-specific data including one or more of: person names, place names, activity types, obj ect names, and dates; generating, by the LLM, text responsive to the prompt based on the task description, the metadata for the one or more images, and the data of the user account, wherein the text is personalized based on the data of the user account and is responsive to the task description; and causing the text to be displayed in a user interface.

12. The computing device of claim 1 1, wherein the operations further comprise: generating respective captions for the one or more images using an image captioning model; and providing the captions to the LLM, wherein the generating the text by the LLM is further based on the respective captions.

13. The computing device of claim 11, wherein the one or more images are in an image album, and the operations further comprise: receiving a user request for an album title for the image album; and in response to the user request, selecting the task description that specifies that the text is for use as the album title, wherein the text includes one or more suggested album titles.

14. The computing device of claim 11, wherein the operations further comprise: receiving a search query from a user; and identifying the one or more images from the image library associated with the user account that match the search query, wherein the user interface includes at least one of the one or more images.

15. The computing device of claim 11, wherein the operations further comprise: receiving user input text; andproviding the user input text to the LLM, wherein the generating the text by the LLM is based at least in part on the user input text.

16. The computing device of claim 11, wherein the operations further comprise, after causing the text and at least one of the one or more images to be displayed in the user interface: receiving user input text; providing the user input text to the LLM; generating, by the LLM, revised text based at least in part on the user input text; and causing the revised text to be displayed in the user interface.

17. The computing device of claim 1 1, wherein the operations further comprise accessing a knowledge repositor}7based on the metadata associated with the one or more images, to obtain additional information, and wherein generating the text is further based on the additional information.

18. A non- transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising: providing a prompt to a large language model (LLM), wherein the prompt includes: a task description; and respective metadata associated with one or more images, wherein the one or more images are from an image library associated with a user account; providing, to the LLM, data of the user account, wherein the data includes userspecific data including one or more of: person names, place names, activity7types, object names, and dates; generating, by the LLM, text responsive to the prompt based on the task description, the metadata for the one or more images, and the data of the user account, wherein the text is personalized based on the data of the user account and is responsive to the task description; and causing the text to be displayed in a user interface.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions cause the processor to perform further operations comprising: generating respective captions for the one or more images using an image captioning model; and providing the captions to the LLM, wherein the generating the text by the LLM is further based on the respective captions.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions cause the processor to perform further operations comprising: receiving a search query from a user; and identifying the one or more images from the image library associated with the user account that match the search query, wherein the user interface includes at least one of the one or more images.

Images