Method of generating one or more graphical assets, and system and method for generating an asset pipeline for dynamically enhancing user experience of a device

Abstract

A method of generating graphical assets includes mapping, by a unified model 602, one or more modalities 725,727,729,731 to a shared representation space 733. The modalities include data received by t

Description

Technical Field Various embodiments relate to a method of generating one or more graphical assets for dynamically enhancing user experience of a user of a device. Further embodiments relate to a method and a system for generating an asset pipeline for dynamically enhancing user experience of a user of a device. Background Today's automotive landscape faces a multifaceted challenge - one that stems from the ever-expanding functions and capabilities integrated into modern vehicles. As cars evolve into sophisticated hubs of technology and connectivity, the demand for user-friendly and adaptable interfaces has never been more pronounced. However, the status quo presents a persistent issue. Standardized interfaces, often riddled with complexities and uninspired design, dominate the automotive industry, leaving drivers and passengers grappling with inefficiencies and distractions. For example, the issues observed in existing solutions may include: • complex and stagnant existing infotainment systems, • predefined layout constraints on displays, • insufficient user engagement and interactivity, • limited customization options to support personalization for in-vehicle displays, • extensive development resources, time, and costs are required when personalized display customization is needed for each driver. Thus, there is a need for a method and system for generating personalized content within a display system that enhance highly personalized in-car experiences catering to each user's unique preferences, thereby addressing at least the problems mentioned above. The video games industry has often been perceived to focus on displays and visual assets. For example, in one prior publication, a computing system for generating visual assets for video games may include an image segmentation model, a first 3D generation model, and a second 3D generation model. At least one of the first 3D generation model and the second 3D generation model may include a machine-learning model. The computing system may be configured to obtain a plurality of images corresponding to the visual asset, where each image shows a different view of an object to be generated in the visual asset; and orientation data for each image that specifies an orientation of the object in the image. A segmented image may be generated for each image. This may involve processing the image using the image segmentation model to segment distinct portions of the image into one or more classes of a predefined set of classes. For each image, 3D shape data may be generated for a portion of the object displayed in the image. This may encompass processing the segmented image of the image, the orientation data of the image, and style data for the visual asset using the first 3D generation model. 3D shape data may be generated for the visual asset. This may include processing the generated 3D shape data of each image using the second 3D generation model. Personalization is often associated with levels of user-friendliness. For example, in one prior publication on personalized speech to video with 3D skeleton regularization and expressive body poses, embodiments have been provided for converting a given speech audio or text into a photo-realistic speaking video of a person with synchronized, realistic, and expressive body dynamics. 3D skeleton movements may be generated from the audio sequence using a recurrent neural network, and an output video may be synthesized via a conditional generative adversarial network. To make movements realistic and expressive, the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic gestures may be embedded into the generation process in both learning and testing pipelines. The former may prevent the generation of unreasonable body distortion, while the latter may help the model quickly learn meaningful body movement with several videos. To produce photo-realistic and high-resolution video with motion details, a part-attention mechanism may be inserted in the conditional generative adversarial network (GAN), where each detailed part may be automatically zoomed in to have their own discriminators. Another prior publication discusses a method for personalizing content such as user interface items provided to a user during a session of an interactive application. One or more environmental context attribute values for the session may be determined. Responsive to a request for content, for at least one item of the content, each of the environmental context attribute values may be mapped to respective values indicating a user’s level of interest in the item for the environmental context attribute values. Content may be retrieved from a content database and the content may be personalized as a function of the user’s level of interest in the item for the environmental context attribute values before being returned to the application. In a prior publication US20230245651A1, an approach for enabling contextually relevant conversational interaction is discussed. Environment data may be received by an Al System which detects a plurality of physical objects in a physical environment and forms a contextual understanding of the plurality of physical objects and the physical environment and identifies a user relevant to the contextual understanding. Summary According to an embodiment, a method of generating one or more graphical assets for dynamically enhancing user experience of a user of a device is provided. The method may include mapping, by a unified model, one or more modalities to a shared representation space, wherein the one or more modalities comprise data received by the unified model from one or more external resources; processing, by a large language model, the shared representation space to generate one or more prompts, wherein the large language model is in communication with the unified model; and generating, based on the generated one or more prompts, the one or more graphical assets for selection to create a personalized user interface for the user of the device. According to an embodiment, a method for generating an asset pipeline for dynamically enhancing user experience of a user of a device is provided. The method may include incorporating an intent or a prompt, and design files retrieved from a storage to generate a design file, incorporating a predetermined layout to the generated design file; retrieving one or more graphical assets generated using a method of generating one or more graphical assets according to an embodiment; optionally, providing access to the retrieved one or more graphical assets for applications available on the device; positioning the retrieved one or more graphical assets within the predetermined layout; and subsequently publishing the predetermined layout on the device to create a personalized user interface for the user. The intent or the prompt may be extracted from at least one of user preferences of the user, activity logs of the users, or specifications of the device. The design files retrieved from the storage may include design files associated with the user and / or the device. According to an embodiment, a system for generating an asset pipeline for dynamically enhancing user experience of a user of a device is provided. The system may include a graphical asset generation module configured to perform a method of generating one or more graphical assets according to an embodiment to geenrate one or more graphical assets; a design file generation module; a graphical asset engine; and the device including an interactive display unit. The design file generation module may include a retriever configured to retrieve design files from a storage; and a generator configured to incorporate an intent or a prompt, and the retrieved design files to generate a design file, and incorporate a predetermined layout to the generated design file, wherein the intent or the prompt is extracted from at least one of user preferences of the user, activity logs of the users, or specifications of the device, and wherein the design files retrieved from the storage may include design files associated with the user and / or the device. The graphical asset engine may be configured to retrieve the generated one or more graphical assets and position the retrieved one or more graphical assets within the predetermined layout; and subsequently provide the predetermined layout to the device. The graphical asset engine may be configured to optionally provide access to the retrieved one or more graphical assets for applications available on the device. The interactive display unit may be configured to publish the provided predetermined layout to create a personalized user interface for the user, and optionally receive inputs and / or feedback signals from the user. Brief Description of the Drawings In the drawings, like reference characters generally refer to like parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which: FIG. 1 shows a flow chart illustrating a method of generating one or more graphical assets for dynamically enhancing user experience of a user of a device, according to various embodiments. FIG. 2 shows a flow chart illustrating a method for generating an asset pipeline for dynamically enhancing user experience of a user of a device, according to various embodiments. FIG. 3 shows a schematic view of a system for generating an asset pipeline for dynamically enhancing user experience of a user of a device, according to various embodiments. FIG. 4 shows a schematic representation illustrating an overview of design file generation, according to an example. FIG. 5 shows a schematic representation of the design file generation of FIG. 4, with details of the generative Al engine for design attributes generation. FIG. 6 shows a schematic representation illustrating an Al-generated assets pipeline, according to an example. FIG. 7 shows a schematic representation of the Al-generated assets pipeline of FIG. 6 in part, with details of the generative Al engine for asset generation. FIG. 8 shows a schematic representation illustrating displays before (left) and after (right) dynamically enhancing user experience, according to an example. Detailed Description The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. Other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. Embodiments described in the context of one of the methods or devices are analogously valid for the other methods or devices. Similarly, embodiments described in the context of a method are analogously valid for a device, and vice versa. Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments. Features that are described in the context of an embodiment may correspondingly be applicable to the other embodiments, even if not explicitly described in these other embodiments. Furthermore, additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments. In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements. In the context of various embodiments, the phrase “at least” may include “exactly” and a reasonable variance. In the context of various embodiments, the term “about” or “approximately” as applied to a numeric value encompasses the exact value and a reasonable variance. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. As used herein, the phrase of the form of “at least one of A or B” may include A or B or both A and B. Correspondingly, the phrase of the form of “at least one of A or B or C”, or including further listed items, may include any and all combinations of one or more of the associated listed items. As used herein, the expression “configured to” may mean “constructed to” or “arranged to”. Various embodiments may provide a method and system for personalized in-car display user interface utilizing generative Al and context understanding. Personalization refers to a system ability to adapt content, experiences, or functionality based on user characteristics. Generative Al, a form of artificial intelligence, emulates human-like content generation. By merging user context (e.g. preferences, season) with generative Al, a solution on personalized user interfaces without the need for designers or developers may be dynamically created. This seamless integration may extend to both driver and center stack displays, immersing users in a driving experience beyond imagination. This innovation leverages customized prompt generation using data from vehicle sensors and user preferences. Personalized content may be subsequently generated within the display system. While this solution may be described with reference to the automotive industry, it may be extended to other fields and applications, e.g. in the entertainment industry involving customized content creation for streaming services, video games, mobile devices, in-flight entertainment, augmented / virtual reality experiences, amongst others, and in the digital marketing industry for creating consistent, compelling online marketing assets that may help to engage the audience. FIG. 1 shows a flow chart illustrating a method 100 of generating one or more graphical assets for dynamically enhancing user experience of a user of a device, according to various embodiments. With reference to FIG. 1, at Step 102, one or more modalities may be mapped, by a unified model, to a shared representation space. The one or more modalities may include data received by the unified model from one or more external resources. At Step 104, the shared representation space may be processed, by a large language model, to generate one or more prompts. The large language model may be in communication with the unified model. At Step 106, the one or more graphical assets may be generated, based on the generated one or more prompts, for selection to create a personalized user interface for the user of the device. In the context of various embodiments, the external resources may include various sensors monitoring the environment and / or objects surrounding the user, user inputs, user actions, user speeches, amongst others. The expression “shared representation space ” essentially refers to a single common space that encompasses the data (the one or more modalities) or derivative(s) thereof. The term “mapping" may mean arranging, translating, or transferring. In various embodiments, mapping the one or more modalities to the shared representation space at Step 102 may include receiving, by a language model, a sequence of input tokens representing the one or more modalities; and generating, by the language model based on the sequence of input tokens, a sequence of output tokens representative of a scenario description within the shared representation space. The shared representation space may include the scenario description(s) based on the mapped one or more modalities. Advantageously, the single, integrated model (language model) used in the method 100 may facilitate the interpretation of data from all exterior sensors, producing scenario descriptions in textual format. Moroever, the method 100 for generating prompts (e.g. Step 106 of FIG. 1) may rely on user preferences and scenario descriptions to steer the creation of graphical assets and visual user interface (UI) elements. The language model may include an encoder-decoder transformer. For example, the encoder-decoder transformer may include one or more modality-specific encoders pre-trained with datasets related to the one or more modalities. The language model may be optimized using multimodal curriculum learning to perform a conditional next token prediction. Generating the sequence of output tokens may include applying the conditional next token prediction on the sequence of input tokens. In other embodiments, mapping the one or more modalities to the shared representation space at Step 102 may include using a dataset including a plurality of image-Mx pairs fortraining, where x = 0,1, 2,..., and Mx represents each modality of the one or more modalities; for each of the plurality of image-Mx pairs, obtaining a corresponding observation in Mx; encoding each image and the corresponding observation (e.g. based on a transformer architecture) into normalized embeddings; and optimizing the normalized embeddings within the shared representation space to align pairs of Mx. In other words, the dataset may be in the form of (I, Mx) where I represents image and Mx represents another modality which may be text (e.g. Mi), audio (e.g. M2), depth (e.g. M3), and so on. The method 100 may use pairs of modalities (I, Mx) to learn a single joint embedding. The dataset containing (image, text) pairs along with naturally occurring image pairings with other modalities such as audio, depth, and so on are used for training. Each image and its corresponding observation in another modality may be encoded into normalized embeddings using deep neural networks. The normalized embeddings and encoders (performing the encoding) may be optimized to make image and other modalities embeddings closer in the joint embedding space. In this way, a single joint embedding space is attained, allowing align pairs of modalities (e.g. Mx, Mx+y) even though training is performed using the pairs (I, Mx), (I, Mx+y), where y = 1, 2,.... For example, a pair of modalities (e.g. Mi, M2) may be aligned based on the training using the pairs (I, Mi), (I, M2), while another pair of modalities (e.g. Mi, M3) may be aligned based on the training using the pairs (I, Mi), (I, M3), and so on. Optimizing the normalized embeddings (or the encoding step) may be performed using a loss function, preferably information noise-contrastive estimation. In an embodiment, processing the shared representation space to generate the one or more prompts at Step 104 may include collectively processing a scenario description and preferences of the user to generate the one or more prompts. In one embodiment, at Step 106, when the generated one or more prompts include a first keyword of “title”, “heading”, “name”, or “caption” or a first term (expression / word / phrase) related to the first keyword, the one or more graphical assets including a task-related text may be generated by a text-to-text model based on the generated one or more prompts. The text-to-text model may introduce to the generated one or more prompts a prefix corresponding to a task to form the task-related text. The task may include translation, title generation or caption generation. In an embodiment, at Step 106, when the generated one or more prompts include a second keyword of “image” or a second term (expression / word / phrase) related to the second keyword, the one or more graphical assets including a raster image may be generated by a text-to-image model based on the generated one or more prompts. The text-to-image model may include a frozen-text encoder to encode the generated one or more prompts into text embeddings, and may map the text embeddings to the raster image. In an embodiment, at Step 106, when the generated one or more prompts include a third keyword of “video” or a third term (expression / word / phrase) related to the third keyword, the one or more graphical assets including a video may be generated by a text-to-video model based on the generated one or more prompts. The video may be based on spatiotemporal layers and optionally, a frame interpolation network. The text-to-video model may include the text-to-image model to generate a plurality of raster images represented spatially and temporally, and a convolutional attention mechanism applicable to the plurality of raster images to obtain the spatiotemporal layers that include spatiotemporal convolution and attention layers. In an embodiment, at Step 106, when the generated one or more prompts include a fourth keyword of “3D” or a fourth term (expression / word / phrase) related to the fourth keyword, the one or more graphical assets including a 3D object may be generated by a text-to-3D model based on the generated one or more prompts. The text-to-3D model may include an untrained Neural Radiance Fields model to predict an initial rendering based on a descriptive caption, may use the text-to-image model based on the descriptive caption and the initial rendering to generate a first image, may remove noise from the first image to obtain a second image for enhancement of the Neural Radiance Fields model, and may perform iterations until the 3D object is obtained. In an embodiment, at Step 106, when the generated one or more prompts include a fifth keyword of “music”, or “sound” or a fifth term (expression / word / phrase) related to the fifth keyword, the one or more graphical assets inclduing an audio waveform may be generated by a text-to-audio model based on the generated one or more prompts. The text-to-audio model may be pre-trained to provide audio representations and text representations, may extract a first set of audio tokens from the audio representations and text tokens from the text representations, may predict semantic tokens using the first set of audio tokens as conditioning, may predict acoustic tokens from the first set of audio tokens and the semantic tokens, may generate a second set of audio tokens based on the semantic tokens, the acoustic tokens and the text tokens as conditioning, and may transform, e.g. by a decoder, the second set of audio tokens into the audio waveform. In various embodiments, the method 100 may further include storing the one or more graphical assets (obtained from Step 106) in a chronological date-time order. FIG. 2 shows a flow chart illustrating a method 220 for generating an asset pipeline for dynamically enhancing user experience of a user of a device, according to various embodiments. With reference to FIG. 2, at Step 222, an intent or a prompt or both, and documentations may be incorporated to generate a design file. The documentations for incorporated with the intent or prompt may be retrieved from a storage (or a memory device). The documentations retrieved from the storage may include documents associated with creating the design file. In other words, the storage may hold a collection of documents related to the creation of design files. This documentation pool acts as an external data store that may be regularly updated with new content (e.g. documentation for newly released libraries or updated display specifications, amongst others) without the need to retrain any model components. User, device, or other information need not be included in this documentation pool. The intent or the prompt may be extracted from at least one of user preferences of the user, activity logs of the users, or specifications of the device. At Step 224, a predetermined layout may be incorporated to the generated design file. At Step 226, one or more graphical assets generated using a method for generating one or more graphical assets may be retrieved. At Step 230, the retrieved one or more graphical assets may be positioned or arranged within the predetermined layout. At Step 232, the predetermined layout may be subsequently published on the device to create a personalized user interface for the user. Optionally, at Step 228, access to the retrieved one or more graphical assets may be provided for applications available on the device. For example, such access may provided to third-party applications via application programming interface (API). With this, the third-party applications may seamlessly incorporate graphical assets or visual elements into their content, thereby elevating the overall user experience. The method for generating the one or more graphical assets used at Step 226 may include or involve the same or like elements or components as those of the method 100 of FIG. 1, and as such, the like elements may be as described in the context of the method 100 of FIG. 1, and therefore the corresponding descriptions are omitted here. The method 220 may further include receiving feedback signals from the user interacting through the device; and feeding the feedback signals into the large language model (reference to Step 104 of FIG. 1) used in the method (e.g. 100, FIG. 1) for processing to adjust the one or more prompts for generating the one or more graphical assets. User feedback signals may come into play whenthea user expresses dissatisfaction with the generated visual elements or UI content. This feedback may cause prompts to be regenerated to create graphical assets or visual UI elements for the specific situation. Steps 222 and 224 may advantageously provide a personalized design file, encompassing code and configurations, as well as hinging on user preferences, activity logs, and device specifications. The generated design file may dictate UI layout, font choices, frequently used app displays, and more. While each of the method described above is illustrated and described as a series of steps or events, it will be appreciated that any ordering of such steps or events are not to be interpreted in a limiting sense. For example, some steps may occur in different orders and / or concurrently with other steps or events apart from those illustrated and / or described herein. In addition, not all illustrated steps may be required to implement one or more aspects or embodiments described herein. Also, one or more of the steps depicted herein may be carried out in one or more separate acts and / or phases. FIG. 3 shows a schematic view of a system 340 for generating an asset pipeline for dynamically enhancing user experience of a user of a device 352, according to various embodiments As shown in FIG. 3, the system 340 may include a graphical asset generation module 342 configured to perform a method of generating one or more graphical assets to generate the one or more graphical assets; a design file generation module 344; a graphical asset engine 350; and the device 352. In the graphical asset generation module 342, the method for generating the one or more graphical assets may include or involve the same or like elements or components as those of the method 100 of FIG. 1, and as such, the like elements may be as described in the context of the method 100 of FIG. 1, and therefore the corresponding descriptions are omitted here. The design file generation module 344 may include a retriever 348 configured to retrieve documentations from a storage 356; and a generator 346 configured to incorporate an intent or a prompt, and the retrieved documentations to generate a design file, and incorporate a predetermined layout to the generated design file. The intent or the prompt may be extracted from at least one of user preferences of the user, activity logs of the users, or specifications of the device. The documentations retrieved from the storage 356 may include documents associated with creating the design file. For example, the storage 356 may be external to the system 340. In another example (not shown in FIG.3), the storage 356 may form part of the system 340. The graphical asset engine 350 may be configured to retrieve the generated one or more graphical assets; optionally, provide access to the retrieved one or more graphical assets for applications available on the device 352; and position the retrieved one or more graphical assets within the predetermined layout; and subsequently provide the predetermined layout to the device 352. The device 352 may include an interactive display unit 354 configured to publish the provided predetermined layout to create a personalized user interface for the user, and optionally receive inputs and / or feedback signals from the user. The generator 346 and the retriever 348 may be in communication with each other as denoted by a line 362. The retriever 348 and the storage 356 may be in communication with each other as denoted by a dashed line 364. The graphical asset engine 350 may be in communication with the graphical asset generation module 342, the design file generation module 344 and the device 352, as denoted by lines 366, 360, and 368 respectively. The system 340 may perform the method 220 of FIG. 2, thereby including or involving the same or like elements or components as those of the method 220 of FIG. 2, and as such, the like elements may be as described in the context of the method 200 of FIG. 2, and therefore the corresponding descriptions are omitted here. The system 340 and the method 220 may facilitate the implementation of personalized adaptive user interfaces on various devices, including but not limited to mobile phones, AR / VR devices, heads-up displays, smartwatches, and other intelligent gadgets. Examples The method 100 of generating one or more graphical assets (FIG. 1), as well as the method 220 (FIG. 2) and the system 340 (FIG. 3) for generating an asset pipeline for dynamically enhancing user experience of a user of a device may be exemplified in relation to a highly personalized in-car experience catering to users unqiue preferences and needs, as discussed below. Camera images, sensor data, voice commands, and infotainment inputs (including radio) may be utilied to process and generate personalized prompts according to user preferences. Using the given prompt, generative Al-based techniques may be employed to generate a wide range of graphical assets, encompassing captivating images, dynamic videos, audios, and immersive 3D elements. The generated graphical assets may be dynamically applied to restyle various apps present within the infotainment system, including but not limited to typography, color palettes, buttons, animations, and imagery. A customized design file may be created by considering the user profile, user activity log, and device specifications. The design file tailors the screen layout, element placement, and stylescape to match the user's preferences. The produced graphical assets seamlessly integrate into various displays, including the infotainment system, AR / VR system, or windshield, ensuring a harmonious fit. FIG. 4 shows a schematic representation illustrating an overview of design file generation 422, according to an example. FIG. 5 shows a schematic representation of the design file generation 422 of FIG. 4, with details of the generative Al engine 444 for design attributes generation. FIG. 6 shows a schematic representation illustrating an Al-generated assets pipeline 640, according to an example. FIG. 7 shows a schematic representation of the Al-generated assets pipeline 640 in part, with details of the generative Al engine 642 for asset generation. The design file generation 422 and the generative Al engine 444 for design attributes generation may include the same or like elements or components as those of Step 222 of FIG. 2 and the design file generation module 344 of FIG. 3, respectively, and as such, the same ending numerals are assigned and the like elements may be as described in the contexts of Step 222 of FIG. 2 and the design file generation module 344 of FIG. 3, respectively, and therefore the corresponding descriptions may be omitted here. The Al-generated assets pipeline 640 and the generative Al engine 642 for asset generation may include the same or like elements or components as those of the system 340 of FIG. 3 and the graphical asset generation module 342 of FIG. 3, respectively, and as such, the same ending numerals are assigned and the like elements may be as described in the contexts of the system 340 of FIG. 3 and the graphical asset generation module 342 of FIG. 3, respectively, and therefore the corresponding descriptions may be omitted here. Further details will be described below with reference to FIGS. 4 to 7. A user profile system 401 may be used to capture and store user preferences, interests, likings, and favorites to personalize the user experience. The platform may facilitate the collection of in-depth data regarding user preferences and interests, utilizing both manual input and automated methods through integration with social media. The system 401 may enable drivers (users) to modify their preferences and interests at their convenience. The display unit 654 may present an automated user profile login system utilizing advanced facial recognition technology. Furthermore, this system 401 holds the potential to encompass additional biometric verification methodologies, including fingerprint and voice recognition. Data may be processed from the vehicle-exterior sensors 619 such as infrared sensors, motion sensors, camera sensors, audio sensors, vehicle sensors, biological sensors, wireless signal sensors, amongst others. A unified model 602 may enable the mapping of different modalities (e.g. data from all sensors 619, e.g. in forms of text 725, image 727, audio 729 and / or other forms of data 731) to a shared and more abstract representation space, which may be processed by a large language model (LLM) 735 to generate a detailed textual description. This may be performed either by unification which allows the support of a myriad of tasks and modalities within one unified framework, or by learning a single joint embedding space for all modalities by using images to bind them together. For unification, a language model (LM) may be used as the core model, implementing lightweight modality-specific input projections for unified input / output representations. The LM may accept a sequence of tokens representing different modalities as input and generate a sequence of tokens as output. For example, the architecture of the LM may be a typical encoder-decoder transformer. Modality-specific encoders may be pre-trained using related high-quality datasets. The LM may be optimized for conditional next token prediction through multimodal curriculum learning (MCL). The proficiently trained LM may gain the ability to process multiple sensor inputs and produce textual descriptions. For learning a single joint embedding space for all modalities, an extensive dataset including pairs of (image, text), encompassing a diverse array of semantic concepts may be utilized. An inherent, self-supervised fusion of additional modalities (data from the sensors 619) with corresponding images sourced from camera sensors. When presented with an image and its corresponding observation in the alternative modality, both may be encoded into standardized embeddings. The encoder for all modalities may adopt a Transformer architecture. The optimization of embeddings and encoders may involve an information noise-contrastive estimation (infoNCE) loss, effectively aligning the embeddings of images and the other modalities. This highly trained model may result in spontaneous alignment across all modalities, enabling it to perform a wide range of multimodal tasks across various modalities. A large language model 735 may be adapted to generate a suitable prompt 737a-737e for creating graphical assets based on the user profile 401 and the scenario description 733 obtained from the unified model 602. More specifically, the user profile 401 (user preferences 603), along with the scenario description 733 may be passed as an input (at 749) to the pre-trained LLM 735. The pre-trained LLM 735 may generate several prompts 737a-737e suitable for generating image, video, and 3D graphical assets. In a different example (not shown in the figures), the user preferences (e.g. 603) may instead be one of the inputs, along with the data from the sensors 619, into the unified model 602. Here, the output from the unified model 602 may be expected to generate a personalized scenario description 733 that is directed to the LLM 735. A Text-to-Text model 739 may generate texts using the prompt 737a, e g. containing the keyword “title”, “heading”, “name”, “caption” or any other related terms. A pretrained Text-to-Text model 739 such as T5 may enable the conversion of all text-based language challenges into a text-to-text format. Each task under consideration, whether the task is translation, title generation, or caption generation, may be framed as providing text input to the T5 model and training it to produce specific target text. The pretrained or finetuned model 739 may exhibit strong performance across various tasks by introducing distinct prefixes to the input that correspond to each specific task. For instance, for translation: "translate English to German:..." for title generation: "title: ..." and so forth. A Text-to-lmage model 741 may generate images using the prompt 737b, e.g. containing the keyword "images” or any other related terms. A base Text-to-lmage model 740 may be trained on text-image pairs. A frozen-text encoder may be employed to encode the input text into text embeddings. A cascade of conditional diffusion models may map these embeddings to images of increasing resolutions. The trained model 741 may produce high resolution image samples with unprecedented photorealism and alignment with text. The model 741 may be anticipated to produce raster images. To support scalability, the model 741 may be fine-tuned to convert raster images into vector format, resulting in scalable vector grpahics (SVG) images. Additionally, the model 741 may be refined further through latent score distillation to enhance both quality and coherence. The Text-to-Video model 743 may generate videos using the prompt 737c, e g. containing the keyword “videos” or any other related terms. A base Text-to-lmage model (e.g. 740) may be trained on text-image pairs. Spatiotemporal convolution and attention layers may extend building blocks of the networks to the temporal dimension. A spatiotemporal network may include both spatiotemporal layers and a frame interpolation network for high frame rate generation. The trained model 743 may generate richer content with motion consistency and text correspondence. The Text-to-3D model 745 may generate 3D objects using the prompt 737d, e.g. containing the keyword “3D” or any other related terms. A descriptive caption may be provided and modified as necessary for the desired camera viewpoint. An untrained NeRF (Neural Radiance Fields) model may be used to predict the initial rendering based on the caption. An image may be generated using the Text-to-lmage model 741, guided by both the caption and the initial NeRF rendering. Specific noise may be added to match input requirements of the Text-to-lmage model 741. A higher-quality image may be created with the Text-to-lmage model 741, while preserving the generated image but removing added noise. The resulting higher-quality image may be used to guide and enhance the NeRF model for improved results. The entire process may be repeated until the 3D model achieves the desired level of quality, adjusting as needed. In other words, this noise is designed to align the distribution of the rendering with the expected input distribution of the Text-to-lmage model 741. The noise added to the NeRF rendering is sampled from a normal distribution with the same mean and standard deviation as the noise used to train the Text-to-lmage model 741. This ensures that the noise characteristics are compatible with the training data of the Text-to-lmage model 741. The noise addition process is designed to be differentiable, allowing gradients to flow back from the Text-to-lmage model 741 to the NeRF parameters during optimization. This enables the NeRF to be refined based on the feedback from the generated images. The noisy NeRF rendering is then fed into the Text-to-lmage model 741, which generates a 2D image based on the caption and the initial rendering. This generated image is used to further refine the NeRF, and the process iterates until the 3D model meets the desired quality standards. The trained model 745 has the capability to produce a 3D model based on a given text. This model 745 may be observed from various angles, illuminated by any chosen lighting, or seamlessly integrated into a diverse range of 3D environments. A Text-to-Audio model 747 may generate audios using the prompt 737e, e.g. containing the keyword “music”, “sound”, and other musical / sound related terms. Independent pretraining of models such as Soundstream, w2v-BERT, and MuLan may be used to provide the audio and text representations. During training, MuLan audio tokens , semantic tokens, and acoustic tokens may be extracted from the audio-only training set. In the semantic modeling stage, semantic tokens may be predicted using MuLan audio tokens as conditioning. In the subsequent acoustic modeling stage, acoustic tokens may be predicted, given both MuLan audio tokens and semantic tokens. Each stage may be modeled as a sequence-to-sequence task using decoder-only Transformers. The trained model 747 may possess the ability to utilize MuLan text tokens, which are derived from the text prompt 737e, as a conditioning signal. The model 747 may then transform the generated audio tokens into waveforms, e.g. through the Soundstream decoder. Text, Images, Videos, Audios, and 3D assets respectively generated from the Text-to-Text model 739, the Text-to-lmage model 741, the Text-to-Video model 743, the Text-to-3D model 745 and the Text-to-Audio model 747 may be stored in a memory storage (or asset storage 617), organized under a datetime folder. The generation of these assets may be described in similar context to the method 100 of FIG. 1. The display unit 654 may feature a preconfigured layout containing a designated space for visual elements. Depending on the established frequency, the most recent graphical assets 621 may be selected by the graphical asset engine 650 to dynamically generate the user interface with the corresponding assets 621 on the display unit 654. The design file 409 may incorporate predetermined layouts. The graphical asset engine 650 may retrieve graphical assets 621 from the asset storage 617 and may position them within designated design file placeholders 615 in the layouts. The graphical asset engine 650 may also facilitate access to these graphical assets 621 for applications (app 613) through API calls. The layout may be customizable, allowing for the positioning of visual elements, including frequently used apps 613, as well as for the option to hide specific elements if not needed. The user's intent 411 or prompt may be extracted from user preferences 403, activity logs 405, and device specifications 407. The generative Al engine 444 for design attributes generation may typically include two modules namely a retriever 448 and a generator 446. The retriever 448 may be responsible for fetching a collection of documentations / pertinent design files from storage 456. The generator 446 may create the design file 409 by incorporating the provided prompt / intent 411 and the retrieved documentations / files from storage 456. User feedback signals 623, whether conveyed through voice or on-screen interactions, may be considered, prompting the regeneration of visual elements in response to these inputs. For example, when the user expresses dissatisfaction, the LLM 735 may utilize the set of previously generated prompts as input. However, each prompt may be extended with a keyword, such as 'revise,' 'change,' 'improve,' and so on. The display unit 654 may also be extended to the heads-up display, windshield, augmented reality (AR) devices, virtual reality (VR) devices as the generated graphical assets are scalable to accommodate different resolutions. FIG. 8 shows a schematic representation illustrating displays before dynamically enhancing user experience 851 (left) and after dynamically enhancing user experience 853 (right), according to an example. After dynamically enhancing user experience by performing the method 220 of FIG. 2, or using the system 340 of FIG. 3, or the Al-generated assets pipeline 640 of FIG. 6, the display 853 presents preferential changes to the themes 855, improved arrangements and / or shapes of the buttons 857 and the icons 859. The display 853 may be enhanced from the initial display 851 based on the following. The data at user preferences 403 (FIGS. 4 and 5) may includes the user's preferences, favorites, and interests. For example, the user's favorite pet may be a cat, and his / her favorite colors may be green and blue. At the user activity log 405, the user's frequent actions, such as often used maps, result in displaying the map as the first icon in the output. The design file 409 may be generated based on the user preference (profile) data 403 and the user activity log 405. When the user is driving, information such as a news update, either as text or audio 619 (FIG. 6), could inform that today is "World Pet Day". The information is then processed by the unified model 602 (FIG. 7), producing a scenario description output 733 (FIG. 7) that states "Today is World Pet Day." The scenario description output 733 and the user preference data 603 (FIG. 7) are combined (at 749, FIG. 7) and passed as input to the large language model (LLM) 735 for processing. The LLM 735 generates prompts and provides input to various Text-to-X models 737a-737e. In asset generation, the outputs from each Text-to-X model 739, 741, 743, 745, 747 are stored in the asset storage 617. A new or existing design file 409 may be updated with the latest graphical assets and then displayed by the display unit 654, with the enhanced display 853. While the invention has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the 5 invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

1. A method of generating one or more graphical assets for dynamically enhancing user experience of a user of a device, the method comprising:mapping, by a unified model, one or more modalities to a shared representation space, wherein the one or more modalities comprise data received by the unified model from one or more external resources;processing, by a large language model, the shared representation space to generate one or more prompts, wherein the large language model is in communication with the unified model; andgenerating, based on the generated one or more prompts, the one or more graphical assets for selection to create a personalized user interface for the user of the device.

2. The method as claimed in claim 1, wherein mapping the one or more modalities to the shared representation space comprises:receiving, by a language model, a sequence of input tokens representing the one or more modalities; andgenerating, by the language model based on the sequence of input tokens, a sequence of output tokens representative of a scenario description within the shared representation space.

3. The method as claimed in claim 2, wherein the language model comprises an encoder-decoder transformer.

4. The method as claimed in claim 3, wherein the encoder-decoder transformer comprises one or more modality-specific encoders pre-trained with datasets related to the one or more modalities.

5. The method as claimed in any one of claims 2 to 4, whereinthe language model is optimized using multimodal curriculum learning to perform a conditional next token prediction; andgenerating the sequence of output tokens comprises applying the conditional next token prediction on the sequence of input tokens.

6. The method as claimed in claim 1, wherein mapping the one or more modalities to the shared representation space comprises:using a dataset comprising a plurality of image-Mx pairs for training, where x = 0, 1,2,..., and Mx represents each modality of the one or more modalities;for each image of the plurality of image-Mx pairs, obtaining a corresponding observation in Mx;encoding each image and the corresponding observation into normalized embeddings;optimizing the normalized embeddings within the shared representation space to align pairs of Mx.

7. The method as claimed in claim 6, wherein encoding each image and the corresponding observation comprises encoding based on a transformer architecture.

8. The method as claimed in claim 6 or 7, wherein optimizing the normalized embeddings is performed using a loss function, preferably information noise-contrastive estimation.

9. The method as claimed in claim 2 or 6, wherein processing the shared representation space to generate the one or more prompts comprises collectively processing a scenario description and preferences of the user to generate the one or more prompts.

10. The method as claimed in any one of claims 1 to 9, wherein generating the one or more graphical assets comprises one of the following:when the generated one or more prompts comprise a first keyword of “title”, “heading”, “name”, or “caption” or a first term related to the first keyword, generating, by a text-to-text model based on the generated one or more prompts, the one or more graphical assets comprising a task-related text;when the generated one or more prompts comprise a second keyword of “image” or a second term related to the second keyword, generating, by a text-to-image model based on the generated one or more prompts, the one or more graphical assets comprising a raster image;when the generated one or more prompts comprise a third keyword of “video” or a third term related to the third keyword, generating, by a text-to-video model based on the generated one or more prompts, the one or more graphical assets comprising a video based on spatiotemporal layers;when the generated one or more prompts comprise a fourth keyword of “3D” or a fourth term related to the fourth keyword, generating, by a text-to-3D model based on the generated one or more prompts, the one or more graphical assets comprising a 3D object; orwhen the generated one or more prompts comprise a fifth keyword of “music”, or “sound” or a fifth term related to the fifth keyword, generating, by a text-to-audio model based on the generated one or more prompts, the one or more graphical assets comprising an audio waveform.

11. The method as claimed in claim 10,wherein the text-to-text model introduces a prefix corresponding to a task to the generated one or more prompts to form the task-related text, the task comprising translation, title generation or caption generation;wherein the text-to-image model comprises a frozen-text encoder to encode the generated one or more prompts into text embeddings, and maps the text embeddings to the raster image;wherein the text-to-video model comprises:the text-to-image model to generate a plurality of raster images represented spatially and temporally, anda convolutional attention mechanism applicable to the plurality of raster images to obtain the spatiotemporal layers comprising spatiotemporal convolution and attention layers;wherein the text-to-3D model comprises an untrained Neural Radiance Fields model to predict an initial rendering based on a descriptive caption, uses the text-to-image model based on the descriptive caption and the initial rendering togenerate a first image, removes noise from the first image to obtain a second image for enhancement of the Neural Radiance Fields model, and performs iterations until the 3D object is obtained; andwherein the text-to-audio model is pre-trained to provide audio representations and text representations, extracts a first set of audio tokens from the audio representations and text tokens from the text representations, predicts semantic tokens using the first set of audio tokens as conditioning, predicts acoustic tokens from the first set of audio tokens and the semantic tokens, generates a second set of audio tokens based on the semantic tokens, the acoustic tokens and the text tokens as conditioning, and transforms the second set of audio tokens into the audio waveform.

12. The method as claimed in any one of claims 1 to 11, further comprising storing the one or more graphical assets in a chronological date-time order.

13. A method for generating an asset pipeline for dynamically enhancing user experience of a user of a device, the method comprising:incorporating an intent or a prompt, and documentations retrieved from a storage to generate a design file,wherein the intent or the prompt is extracted from at least one of user preferences of the user, activity logs of the users, or specifications of the device, andwherein the documentations retrieved from the storage comprise documents associated with creating the design file;incorporating a predetermined layout to the generated design file;retrieving one or more graphical assets generated using a method as claimed in any one of claims 1 to 12;optionally, providing access to the retrieved one or more graphical assets for applications available on the device;positioning the retrieved one or more graphical assets within the predetermined layout; andsubsequently publishing the predetermined layout on the device to create a personalized user interface for the user.

14. The method as claimed in claim 13, further comprising:receiving feedback signals from the user interacting through the device; and feeding the feedback signals into the large language model used in the method as claimed in any one of claims 1 to 12 for processing to adjust the one or more prompts for generating the one or more graphical assets.

15. A system for generating an asset pipeline for dynamically enhancing user experience of a user of a device, the system comprising:a graphical asset generation module configured to perform a method as claimed in any one of claims 1 to 12 to generate one or more graphical assets;a design file generation module comprising:a retriever configured to retrieve documentations from a storage; and a generator configured to incorporate an intent or a prompt, and the retrieved documentations to generate a design file, and incorporate a predetermined layout to the generated design file, wherein the intent or the prompt is extracted from at least one of user preferences of the user, activity logs of the users, or specifications of the device, and wherein the documentations retrieved from the storage comprise documents associated with creating the design file;a graphical asset engine configured to retrieve the generated one or more graphical assets; optionally, provide access to the retrieved one or more graphical assets for applications available on the device; and position the retrieved one or more graphical assets within the predetermined layout; and subsequently provide the predetermined layout to the device; andthe device comprising an interactive display unit configured to publish the provided predetermined layout to create a personalized user interface for the user, and optionally receive inputs and / or feedback signals from the user.29

Images