Personalized content assistance using generative artificial intelligence

Abstract

A method is provided for generating personalized content assistance using generative artificial intelligence (AI). The method includes creating (1400) a first user-content interaction data structure (UCIDS) using user interaction information associated with a user interacting with a first content source, wherein the first UCIDS indicates at least one estimated region of struggle (RoS) for the user; mapping (1402) the first UCIDS to a knowledge graph based on the first content source; determining (1404), using the mapping, at least one subgraph of the knowledge graph associated with the at least one estimated RoS; generating (1406), using the at least one subgraph of the knowledge graph, a first input for one or more generative AI system(s); and 0 sending (1408) the first input to the one or more generative AI system(s).

Description

PERSONALIZED CONTENT ASSISTANCE USING GENERATIVE ARTIFICIAL INTELLIGENCETECHNICAL FEATURES

[0001] The present disclosure is related to communication systems and more particularly to personalized content assistance using generative artificial intelligence.BACKGROUND

[0002] Personalized content assistance can help mitigate information overload and / or other comprehension issues by tailoring the delivery of information to individual preferences and circumstances. Such a personalized approach may take into account various factors (e.g., stress levels, time of day, language barrier(s), etc.) and help users manage and consume content more efficiently and effectively.

[0003] Generative artificial intelligence (Al), a branch of artificial intelligence, can encompass the capability to generate diverse content such as text, code, images, video, voice, and music from a set of input parameters, typically in text form. Large language models (LLMs), a prominent class of generative Al, can utilize natural language processing to communicate in a human-like manner. LLMs and multi-modal LLMs (MLLMs) may be trained on massive datasets (e.g., using unsupervised or supervised learning), enabling them to identify patterns in unstructured data. Various LLMs (e.g., generative pre-trained transformers (GPTs)) may utilize transformer architecture, which can use parallel attention modules to learn associations between objects and concepts. Knowledge graphs may further enhance these models by representing how entities (e.g., things, situations, concepts, etc.) relate to each other, thus enriching the Al's ability to generate coherent and contextually relevant content.SUMMARY

[0004] In some approaches, generative AIs (e.g., an LLM or MLLM) can provide useful information to users in one or more modalities (e.g., ways or modes in which something is expressed, like text, videos, and / or audio). For example, a user may ask a question about a topic or a series of topics and, in response, a generative Al system may provide an answer or set of answers. However, challenges can arise when users attempt to consume or understand received information, especially when the information provided is dense, confusing, or difficult to understand.

[0005] Certain aspects of the disclosure and related embodiments may provide solutions to these or other challenges. In some embodiments, a method is provided (e.g., performed by a device, a node, or a system) for generating personalized content assistance using generative artificial intelligence (Al). The method includes creating a first user-content interaction data structure (UCIDS) (e.g., a spatial graph or list(s)) using user interaction information associated with a user interacting with a first content source, wherein the first UCIDS indicates at least one estimated region of struggle (RoS) for the user. The method further includes mapping the first UCIDS to a knowledge graph (e.g., a representation indicating relationships between things, data, concepts, etc.) based on the first content source, e.g., a book, a website, a video, an audio clip, a data file, a dataset, etc. The method further includes determining, using the mapping, at least one subgraph of the knowledge graph associated with the at least one estimated RoS. The method further includes generating, using the at least one subgraph of the knowledge graph, a first input for one or more generative Al system(s). The method further includes sending the first input to the one or more generative Al system(s).

[0006] In some embodiments, a method is provided (e.g., performed by a device, a node, or a system) for determining subgraph(s) from knowledge graph(s) using additional information, e.g., a second UCIDS, historical RoS data, etc. The method includes creating a second UCIDS using second user interaction information associated with a user interacting with a second content source; wherein creating the second UCIDS includes using data from the first UCIDS or historical RoS data. The method further includes mapping the second UCIDS to the knowledge graph. The method further includes determining, using the mapping of the second UCIDS to the knowledge graph, the at least one subgraph or a different subgraph of the knowledge graph.

[0007] In some embodiments, a method is provided (e.g., performed by a device, a node, or a system) for generating or adjusting a content modality preference (e.g., a user-specific preference indicating a format or modality (e.g., text, images, audio, and / or video) for expressing a particular concept or content). The method includes receiving a response from one or more generative Al system(s). The method further includes parsing the response to obtain one or more concepts to present to a user. The method further includes for each of the concepts that lacks a modality preference in a user-specific modality present data set: selecting, from a predefined set of associations between concepts and modality preferences, a modality preference for presenting the concept; presenting the concept based on the modality preference; determining, based on user feedback, whether the user is satisfied with the presentation of the concept; if the user is deemed satisfied with the presentation of the concept, adding the modality preference to the user-specific modality present data set; and if the user is deemed unsatisfied with the presentation of the concept,performing an iterative process involving presenting the concept using a different candidate modality preference and determining user satisfaction with the new presentation until the user is deemed satisfied and the candidate modality preference is added to the user-specific modality present data set or until all candidate modality preferences have been evaluated or utilized (e.g., tested).

[0008] According to some embodiments, a system, a device, a node, a computer program, a computer program product, a host, a system, or a non-transitory computer-readable medium is provided to perform one of the above methods.

[0009] Certain aspects of these embodiments may provide technical advantages. Various embodiments described herein include a system, a process, a mechanism, or a technique that provides flexible integration with various user interaction with content sources (e.g., documents); strong ability for personalizing content assistance for different users with different preferences or capabilities in using digital content; adaptation to user preferences (e.g., by learning the types of output modality preferred and dynamically generating response based on preferred modalities); adaptation of personalized content assistance via user preferences and learned refinement (e.g., from the experience of a user or similar users) for areas or regions of struggle (RoS) (e.g., concepts and concept relationships where the user struggles the most); flexibility in on-the-fly or dynamic application of personalized content assistance for various content (particularly without prior knowledge about the content); and flexible integration and interaction with external knowledge graphs and different types of generative Al system(s), such as large foundation models (e.g., LLMs, MLLMs, large multi-modal models (LMMM)) or generative adversarial networks (GANs).BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of inventive concepts. In the drawings:

[0011] FIG. 1 is a block diagram illustrating an example of a personalizer system for providing content assistance in accordance with some embodiments;

[0012] FIG. 2 is a flow chart illustrating example aspects of a personalizer system in accordance with some embodiments;

[0013] FIG. 3 is a block diagram illustrating an example content source in accordance with some embodiments;

[0014] FIG. 4 is a block diagram illustrating an example knowledge graph (KG) based on a content source in accordance with some embodiments;

[0015] FIG. 5 is a block diagram illustrating aspects associated with graph reasoning in accordance with some embodiments;

[0016] FIG. 6 is a block diagram illustrating aspects associated with KG extraction and augmentation for multi-modal content in accordance with some embodiments;

[0017] FIG. 7 is a block diagram illustrating aspects associated with obtaining user interaction with a content source in accordance with some embodiments;

[0018] FIG. 8 is a block diagram illustrating aspects associated with building a user-content interaction data structure (UCIDS) in accordance with some embodiments;

[0019] FIG. 9 is a flow chart illustrating example procedures for generating content assistance in accordance with some embodiments;

[0020] FIG. 10 is a flow chart illustrating an example procedure for textualizing a UCIDS and extracting concepts and concept relations therefrom in accordance with some embodiments;

[0021] FIG. 11 is a flow chart illustrating an example procedure for mapping a UCIDS to a KG in accordance with some embodiments;

[0022] FIG. 12 is a flow chart illustrating an example procedure for adapting a content modality preference and creating a response based on the preference in accordance with some embodiments;

[0023] FIG. 13 is a flow chart illustrating an example procedure for refining content modality preferences in accordance with some embodiments;

[0024] FIG. 14 is a flow chart illustrating examples of operations for generating personalized content assistance in accordance with some embodiments;

[0025] FIG. 15 is a flow chart illustrating examples of operations to adjust or modify user experience and / or user preference(s) associated with generating personalized content assistance in accordance with some embodiments;

[0026] FIG. 16 is a flow chart illustrating examples of operations to add or modify a content modality preference in accordance with some embodiments;

[0027] FIG. 17 is a flow chart illustrating examples of operations to map a second UCIDS to an existing KG in accordance with some embodiments;

[0028] FIG. 18 is a block diagram of a device in accordance with some embodiments;

[0029] FIG. 19 is a block diagram of a network node in accordance with some embodiments; and

[0030] FIG. 20 is a block diagram of a virtualization environment in accordance with some embodiments.DETAILED DESCRIPTION

[0031] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present / used in another embodiment.

[0032] Knowledge Graphs (KGs)

[0033] A KG represents how entities, such as things, situations, concepts, relate to each other. A KG may include nodes, edges and labels. Entities or objects are represented by nodes; an edge between two nodes states that there is a relationship between them, and the label on the edge describes details about the relationship. In some embodiments, a KG may be a semantic graph.

[0034] Large Language Models (LLMs) and Generative Artificial Intelligence (Al)

[0035] An LLM (LLM) or a multi-modal LLM (MLLM) can use natural language processing to communicate in a seemingly human way. LLMs may generally focus on generating textual content, while MLLMs may extend this capability by integrating multiple forms of data, enabling the generation of text, images, audio, and other media. An LLM or MLLM may refer to a machine learning model with many parameters used (e.g., 100 or more billion parameters) and is trained on massive amounts of data. To manage this enormous amount of training data, unsupervised learning is commonly employed in the initial phase, allowing the model to recognize patterns in unstructured data. Transformers are a key component of LLMs, utilizing parallel attention modules to establish associations between objects, such as words, and concepts.

[0036] While LLMs and MLLMs can be quite broad in their capabilities and responses, they can be tailored for specific applications through fine-tuning, such as using supervised learning and reinforcement learning from human feedback. Generative Al, a broader category that includes both LLMs and MLLMs, refers to the ability to create various types of content — text, code, images, video, voice, and music — from a set of input parameters, usually provided in text.

[0037] Emotion Recognition (Affective Computing)

[0038] The field of emotion detection and emotion recognition has been researched for several decades, but as for many other fields, the use of machine learning in this field has become popular in the last years. Commonly used modalities for identifying reaction and emotions are byanalyzing visual input, e.g., facial expressions, audio input, written expression but also using other physiological sensor measurements from wearables, e.g., measuring the heart rate, blood pressure and palm sweat. The accuracy of the methods may improve by incorporating inputs from multiple modalities.

[0039] Affective computing is the slightly broader field that, on top of detecting emotions, also includes interpretation and reacting to detected emotions. Typical applications that are foreseen for affective computing include customer service, healthcare, education, entertainment, and gaming.

[0040] Content sources (e.g., documents with images, data files, datasets, etc.) may generally be shared in the same format to all users and are often presented in a non-personalized and overly complex format. Also, not all content sources are well-written, well-organized, or self-contained. As a result, some users might need to spend longer (and often have a more challenging time) to comprehend (e.g., grasp, leam, or master) the content or just to get an overview or the core essence of the content. Learning from content can be a highly individualized process and may be influenced by the user's situation and mood. Effective and timely interaction with content can depend significantly on the user's unique needs and preferences. This personalization is often overlooked in scenarios where a single content format is expected to serve all users.

[0041] Some existing methods attempt to address personalization and adaptive learning in various ways, each with their own limitations. One method provides an interface tool for personalized website content, focusing on content presentation based on user history or preferences, but does not ensure the content's understandability. Another method redefines course delivery in large-scale classes by adapting content presentation based on student learning profiles, but it is limited to a predefined set of course materials. Other methods use large language models for generating automated assistant responses to user queries without considering the user's experience of the content.

[0042] In some embodiments, a system, a device, a node, or a process in accordance with one or more aspects described herein may utilize LLMs or other methods for automating the creation or augmentation of KGs. For example, a specialized generative pre-trained transformer (GPT) may receive a content source (e.g., a digital file, a website with text, images, and videos, or a web document) and optionally other inputs and may generate a KG based on the content source.

[0043] In some embodiments, a system, a device, a node, or a process in accordance with one or more aspects described herein may facilitate the interaction of a user and a content source by taking into account both the knowledge structure of the content and the user’s feedback, and user needs and preferences (e.g., pain points, content modality preferences, and regions of struggle(RoSs)). In some embodiments, a system, a device, a node, or a process in accordance with one or more aspects described herein may include various technical aspects. Example technical aspects include capturing user interaction with a content source, creating a user-content interaction data structure (UCIDS) (such as a user-content interaction graph (UCIG)), using the UCIDS to identify potential RoSs; mapping UCIDS to a KG based on the content source and external KGs, using the potential RoSs to extract relevant subgraph(s) from KG, and using data from the relevant subgraph(s) as context to prompt or trigger one or more generative Al system(s) (e.g., an LLM) to generate clarifying explanations or questions to assist in resolving a user’s struggle to comprehend content, and mechanisms for adapting and further improving abilities of the system to provide personalized content assistance.

[0044] FIG. 1 is a block diagram illustrating an example of a personalizer system 102 for providing content assistance in accordance with some embodiments. The personalizer system 102 may represent any suitable entity or entities (e.g., a device, a server, or a distributed computing system) in a computing environment 100. The computing environment 100 may include one or more networks (e.g., the Internet, a public network, a private network, an access network, etc.) and various devices and / or nodes.

[0045] In some embodiments, the personalizer system 102 may contain elements or modules that run locally (e.g., on a user's laptop or phone) and elements or modules that run remotely (e.g., on a remote server, a distributed system, or a cloud-based system). In some embodiments, user interaction inputs (e.g., facial recordings, gaze recordings, audio recordings, and / or video recordings related to the user interacting with content) may be collected locally and various Al or machine learning processes (e.g., generating or modifying KGs, UCIDSs, input prompts, etc.) for preparing the personalized content are performed on a remote system with the result being sent back to the user.

[0046] The personalizer system 102 may include various hardware components (e.g., processing circuitry, a memory, a power supply, etc.). The personalizer system 102 or component(s) thereof (e.g., processing circuitry) may be configured (e.g., via software or programming) to perform various operations described herein. For example, the personalizer system 102 may receive user feedback and generate a digital twin for the user comprising a user profile with user preferences. In this example, the personalizer system 102 may use the user profile information along with other data (e.g., user interaction information, knowledge graphs, UCIDS, etc.) to generate personalized prompts or inputs for one or more generative Al system(s), e.g., LLMs or MLLMs that are integrated with the personalizer system 102 or that are separate from the personalizer system 102 (e.g., at a different computing platform or in a different network fromthe personalizer system 102). Continuing with this example, the prompts or inputs may be constructed or designed to obtain content (e.g., especially content that the user struggles to understand) in a format or modality that makes it easier to consume or comprehend.

[0047] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may communicate with one or more generative Al system(s). Examples of a generative Al system include an LLM, an MLLM, an LMMM, an Al taste generator, an Al voice generator, an Al visual (e.g., an image, video, and / or hologram) generator, an Al touch (e.g., haptics) generator, an Al smell generator, or a combination thereof.

[0048] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may create or generate UCIDSs (e.g., a user content interaction graph, UCIG; a data tree; a list, etc.) that indicate how a user interacts with content or concepts therein. For example, a UCIDS may be based on different means of user interaction (e.g., eye / head gaze tracking) with a content source (e.g., a web document with multi-modal data). In this example, the UCIDS may represent or indicate at least one estimated RoS, e.g., regions in the web document where user is predicted to struggle in understanding.

[0049] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may assign a quantity or value that represents an estimated degree of difficulty (DoD) or degree of struggle (DoS) for every identified RoS. For example, when obtaining user interaction information as a user reads or consumes a content source (e.g., a web document with multimedia), the personalizer system 102 or entities(s) thereof may monitor the amount of time a user spends on a region and may use this information along with other data (e.g., user voice feedback indicating difficulty) to compute a DoS for each estimated RoS.

[0050] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may textualize RoSs extracted from a UCIDS, jointly with the assigned DoS values, and apply this information when generating an input prompt for a generative Al system, e.g., an assistant LLM, MLLM, or LMMM. For example, the personalizer system 102 or entities(s) thereof may convert or represent a RoS as text that describes the concept(s) expressed in the region and then may use that information about these concept(s) to provide context in an input prompt such that a receiving LLM will response with particular information and / or modalities for expressing these concepts.

[0051] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may textualize subgraphs of a KG (e.g., subgraphs associated with one or more estimated RoSs obtained from a UCIDS) and utilize the textualized information when generating input for one or more generative Al system(s). For example, the personalizer system102 or entities(s) thereof may identify a subgraph from a KG that represents the relationship between various authors and their literary works. In this example, the subgraph may be textualized to describe how certain themes and styles are interconnected among these literary works. This textualized information can then be used as input for an LLM or MLLM to create personalized reading recommendations or even to generate new literary content inspired by the identified themes and styles.

[0052] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may map a UCIDS to a KG derived from a document source to identify subgraphs in the KG corresponding to RoSs, e.g., topics or concepts that a user is struggling with. For example, the personalizer system 102 or entities(s) thereof may associate portions (e.g., one or more subgraphs) of a KG with estimated RoS(s) indicated by a UCIDS and may assign these portions with relevant DoS values. In this example, when generating input for one or more generative Al system(s), the personalizer system 102 or entities(s) thereof may generate the input that includes data from one or more subgraphs associated with the estimated RoSs.

[0053] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, anode, etc.) may utilize algorithms or techniques for constraining input for one or more generative Al system(s). For example, assuming an LLM or MLLM has an input or maximum length or size of 32 thousand tokens, the personalizer system 102 or entities(s) thereof may apply a heuristic-based method or another approach to control the length of the input or a portion thereof (e.g., a RoS related subgraph of a KG) such that the input does not exceed a permitted context length associated with the LLM or MLLM. For instance, a heuristic-based method may involve ranking the nodes of the knowledge graph or a subgraph thereof based on their relevance to a current query and / or a related DoS and then selecting the top N nodes until the context length limit is reached. In another example, the personalizer system 102 or entities(s) thereof may use a sliding window approach in generating input prompts, e.g., where the window size is adjusted to capture more pertinent information in an input prompt, while discarding less relevant details. By considering a context size or length associated with one or more generative Al system(s), these approaches help maintain the most relevant and meaningful input data within the allowed context length, ensuring effective processing by the one or more generative Al system(s).

[0054] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may be adapted based on user experience. For example, personalizer system 102 or entities(s) thereof may monitor one or more users interacting withvarious content sources and may analyze their interactions (e.g., via gaze tracking, user reactions, emotional analysis, etc.) and using this information may leam which areas or RoSs (e.g., concepts and relation between concepts) that users find difficult to understand, through bootstrapping from past interactions as well as experience of other (similar) users. In some embodiments, by bootstrapping from prior experiences of other similar users, the personalizer system 102 or entities(s) thereof can identify common points of confusion or difficulty and, over time, become more effective in addressing a user's unique learning needs, thereby enhancing the overall user experience. For example, if a user frequently struggles with understanding the relationship between different cat breeds and their genetic predispositions, the personalizer system 102 or entities(s) thereof can tailor input for a generative Al system to obtain responses that provide clearer explanations and additional resources on these topics.

[0055] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may be adapted based on learned content modality preferences. In some embodiments, the personalizer system 102 or entities(s) thereof may identify which modality or modalities (e.g., text, visual, audio) for representing content are most effective for a particular user in terms of efficient usage (e.g., consumption or comprehension) of the content. For example, if a user is having difficulty learning about the relationship between different cat breeds and their dietary needs, the personalizer system 102 or entities(s) thereof may determine that the user comprehends this information more effectively through visual aids. Consequently, the system would generate input that triggers a generative Al system (e.g., an MLLM) to respond with infographics or charts illustrating the dietary needs of various cat breeds, rather than detailed text descriptions. Conversely, if another user prefers textual information for this type of content, the personalizer system 102 or entities(s) thereof may generate input that triggers the generative Al system to provide comprehensive written guidelines and explanations about each breed's dietary requirements. By learning and using content modality preferences, the personalizer system 102 or entities(s) thereof may ensure that a generative Al system provides content in a suitable modality, which can enhance the overall user experience and improve the efficiency with which users engage with the content.

[0056] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may dynamically capture and analyze user interaction with a content source (e.g., a web document). In some embodiments, the personalizer system 102 or entities(s) thereof may gather or obtain user interaction information via various sources, inputs, or approaches. For example, the personalizer system 102 or entities(s) thereof may utilize a user's gaze pattern, sensors measurements, mouse movements, reactions and emotion (e.g., by detectinga measured cognitive load or a facial expression such as frowning), and / or by applying a user's preferences and feedback (e.g., a user saying “I don’t understand this part” and drawing a bounding box to the region of struggle (RoS)), by extrapolating learned past interactions of other users with the content, or by generalizing learned past interactions of the user with other contents (the digital twin of the user) or alike, and a combination of one or more means discussed above.

[0057] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may represent gathered user interaction information associated with a content source in a UCIDS. For example, a generated UCIG or a spatial graph may include nodes that represents different portions of a document and edges that represent transitions or interactions between portions. In this example, each node and edge in a UCIG or spatial graph may be assigned with a quantity or value that represents the DoS associated with the concepts (or concept relationships) contained in the identified RoS. In some embodiments, DoS values are assigned based on the duration of a user’s fixation on a region (e.g., a node’s DoS) in a content source as well the frequency of user’s gaze transitioning from one region to another (e.g., an edge’s DoS).

[0058] In some embodiments, the personalizer system 102 or entities(s) thereof (e.g., a module, a device, a node, etc.) may generate a KG based on a content source and may augment the KG with further information, such as reasoning and / or bootstrapping from relevant external KGs.

[0059] In some embodiments, the personalizer system 102 or entities thereof (e.g., a module, a device, a node, etc.) may generate a KG based on a content source and may augment the KG with additional information, such as reasoning and / or bootstrapping from relevant other KGs. For example, the personalizer system 102 or entities(s) thereof may use reasoning to infer relationships between entities, such as deducing that two historical figures are connected through a common event. In this example, the personalizer system 102 or entities(s) thereof may integrate information from external KGs, such as augmenting a KG based on a medical content source with a public KG representing drug interactions between various medicines and substances to enrich the KG with detailed information about drug interactions. In some embodiments, augmenting a KG using other (e.g., broad knowledge) KGs helps enrich and broaden a KG based on a content source, which, in turn, helps the personalizer system 102 or entities(s) thereof provide effective content assistance.

[0060] In some embodiments, the personalizer system 102 or entities thereof (e.g., a module, a device, a node, etc.) may map a UCIDS to a KG. For example, mapping a UCIDS to a KG may determine subgraph(s) in the KG that are associated with one or more estimated RoSs indicated by the UCIDS. In this example, the mapping may be usable in identifying a context whichencompasses the concepts and concepts relationships corresponding to user’s RoSs and broadens the context with relevant information from the KG. In some embodiments, where a UCIDS is a spatial graph and a KG is a semantic graph, the UCIDS may be transformed into a semantic domain before mapping the UCIDS to the KG.

[0061] In some embodiments, the personalizer system 102 or entities thereof (e.g., a module, a device, a node, etc.) may assist a user by generating personalized input prompts for one or more generative Al system(s). For example, the personalizer system 102 or entities thereof may generate an input prompt for one or more LLMs, each handling one or more modalities, such as GPT-4v or alike. In some embodiments, a generated input prompt may include textualized RoS data, textualized subgraph(s) data derived from mapping a UCIDS to a KG, and optionally the original content source (which may be used for referencing or grounding the LLM responses).

[0062] In some embodiments, the personalizer system 102 or entities thereof (e.g., a module, a device, a node, etc.) may assist a user in grasping content in different ways, such as referencing or grounding generative Al responses with an original content source, representing concepts (and concept relationships) in different modalities (text, image, video, audio), crafting clarifying questions and answers focused on the subset of the content associated with RoSs (e.g., identified via mapping of a UCIDS to a KG) and alike, and by taking into account past experiences of the user and / or similar users consuming (similar) content.

[0063] In some embodiments, the personalizer system 102 or entities thereof (e.g., a module, a device, a node, etc.) may implicitly and / or explicitly observe user feedback regarding the usefulness of the content assistance provided. For example, the system may evaluate whether transforming the information to a visual format was helpful or if the provided explanations, reasonings, and questions were relevant to the user's difficulties in understanding the content. In this example, the user feedback and related analysis may then be applied to adapt the system for enhanced performance in future interactions with the user.

[0064] In some embodiments, user feedback may be direct, such as when a user answers specific feedback questions, or user feedback may be indirect, e.g., by analyzing which types of assistance led to reduced user struggles. Indirect feedback may include observing reduced fixation on certain regions of the content, decreased eye gaze transitions between regions, or an improved emotional state. By incorporating user feedback, the personalizer system 102 or entities thereof may improve its effectiveness and provide more personalized assistance.

[0065] FIG. 2 is a flow chart illustrating an example framework 200 for generating personalized content assistance in accordance with some embodiments. In some embodiments, the framework 200 may represent operations and / or technical aspects that are performed by thepersonalizer system 102 or entities thereof (e.g., a module, a device, a node, etc.) for providing or generating personalized content assistance or other functions.

[0066] Operations of the personalizer system 102 or entities thereof (e.g., implemented using the structure of FIG. 18, 19, or 20) will now be discussed with reference to the flow chart of FIG. 2 according to some embodiments of the present disclosure. For example, modules (e.g., a personalizer module or an input generator) may be stored in memory 1810 of FIG. 18, and these modules may provide instructions so that when the instructions of a module are executed by respective processing circuitry 1802, device 1800 (e.g., user equipment (UE), a server, or a node, etc.) performs respective operations of the flow chart of FIG. 2.

[0067] In some embodiments, operations of the framework 200 may be performed by the personalizer system 102 or entities thereof after a set of prerequisites are met. One prerequisite may involve bootstrapping or leveraging a personal assistant service that personalizer system 102 or entities thereof can utilize or interact with. Another prerequisite may involve deploying or configuring a user interaction detection system to identify RoS(s) in a content source for a user. For example, the user interaction detection system may comprise one or more system(s) for tracking the user’s interaction with the document (e.g., an eye-tracking system, an emotion or stress recognition system, etc.) and may be capable of correlating user-document interaction with determined emotions (e.g., via an emotion recognition system or by correlating emotion while a user is gazing at certain regions of the document). In this example, the user interaction detection system may also include an emotion recognition model that is downloaded and configured for various sensors that monitor user interactions or behaviors. In some embodiments, personalization of the user interaction detection system or aspects thereof (e.g., an emotion recognition system) may be needed and may involve the user explicitly responding to prompts with feedback, e.g., during early stages of use.

[0068] In some embodiments, the framework 200 or related aspects may include optionally bootstrapping or leveraging a user behavioral model that learns the preferences of a user. For example, the personalizer system 102 or entities thereof may use user feedback and user interaction information to modify or refine a digital twin or a related user profile representing a user. In this example, as the user requests additional information or interacts with different content sources, the user behavioral model may take as input the additional user feedback and / or user interaction information to determine content modality preferences or other user preferences.

[0069] In some embodiments, the framework 200 or related aspects may include bootstrapping or leveraging a learned history of RoSs (HROS), e.g., data indicating what topics and relationships that a user or group of similar users has difficulty in grasping. For example, thepersonalizer system 102 or entities thereof may store HROS and use the stored HROS to adapt or refine various aspects associated with generating personalized content assistance. In some embodiments, HROS may be updated during an online phase of the framework 200.

[0070] In some embodiments, an online phase of the framework 200 may involve various operations or actions, e.g., when a user interacts with a content source and input is generated to trigger a response from a generative Al system. In some embodiments, the online phase may include a user downloading or accessing a content source (e.g., a document or story) to a device (e.g., a UE); the device or another entity (e.g., a server) creating a KG of the content sources (e.g., potentially using an LLM, a GPT, additional KGs, or cloud services); the content source being presented to the user; the device (e.g., using sensors or input / output devices) capturing user interaction information (e.g., user reactions, facial expressions, vocal responses, etc.) and analyzing (e.g., using sensor measurements, a user profile, user preferences, a user behavioral model, historical data (e.g., HROS), etc.) the user interaction information to generate or update a UCIDS; the device or another entity (e.g., a server) using (e.g., by mapping the UCIDS to the KG) the UCIDS to extract or identify information (e.g., subgraphs associated with a user’s RoSs) from the KG; the device or another entity (e.g., a server) creating input context (e.g., an input prompt) for a generative Al system (e.g., using information from the UCIDS as well as information extracted from the KG); the generative Al system (e.g., an LLM or MLLM) generates content (e.g., multi-modal content for assisting the user in comprehension and / or clarification) and presents the content to the user; the device or another entity (e.g., a server) observes and analyzes the user’s experience (e.g., whether the user is satisfied with the data presentation) to determine whether to adapt the response from the generative Al system; if the observation indicates a negative user experience (e.g., dissatisfaction with the data presentation), the device or another entity (e.g., a server) utilizes an iterative process where different content modalities are evaluated or utilized until all modalities are exhausted or the user’s experience is deemed positive; and if the observation indicates a positive user experience (e.g., satisfaction with the data presentation), the device or another entity (e.g., a server) updates a user profile, user preferences, content modality preference, or other aspects of the framework 200 based on the feedback / modality used.

[0071] In some embodiments, when a user interacts with a new content source (e.g., a second document), a UCIDS based on this content source may be bootstrapped or leverage with the UCIDS of a prior content source (together with the DoS values indicated by the new UCIDS). Then, the UCIDS of the prior content source may be adapted with respect to the current interaction of the user with the second content source. In some embodiments, the personalizer system 102 orentities(s) thereof may store this information as HROS and may optionally consolidate the experience of similar users in the HROS.

[0072] FIG. 3 is a block diagram illustrating an example content source 300 in accordance with some embodiments. As depicted in FIG. 3, the content source 300 is a story of a cat that is looking for a cozy place to nap during a cold winter day.

[0073] FIG. 4 is a block diagram illustrating an example knowledge graph (KG) 400 based on a content source in accordance with some embodiments. As depicted in FIG. 4, the KG 400 is based on the content source 300 in FIG. 3 and includes various nodes representing various entities or concepts in the story (e.g., the cat’s attributes or traits, the cat’s environment properties, etc.) and connections (e.g., lines or arrows) and related labels representing relations between the items (e.g., has, type of, causes, is made of, is near, is a, likes, etc.).

[0074] FIG. 5 is a block diagram illustrating aspects associated with graph reasoning in accordance with some embodiments. Various operations or aspects referred to in FIG. 5 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., anode, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations or aspects referred to in FIG. 5, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0075] As depicted in FIG. 5, a KG 500 represents an augmented portion of the KG 400. For example, the KG 500 or data therein is generated when the KG 400 is processed via graph reasoning (e.g., a machine learning technique or other technique) to infer implicit facts and / or relationships about nodes (e.g., concepts or entities) in the KG 400. In this example, the concept “Whiskers feels cold” is not directly given by the KG 400 but can be inferred through graph reasoning. As shown in the KG 500, the concept “Whiskers feels cold” is represented by a dotted arrow pointing from the node “Whisker” (the cat) to the node “cold” with the label “feels”.

[0076] FIG. 6 is a block diagram illustrating an example process 600 associated with KG extraction and augmentation for multi-modal content in accordance with some embodiments. For example, augmenting a KG may utilize additional KGs (e.g., external or public KGs) along with derived or inferred data from a graph reasoning process, where the augmented KG ( may contain a much broader or richer KG than the original KG derived from the document. Various operations in the process of FIG. 6 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein maybe configured for performing one or more operations in the process of FIG. 6, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0077] Referring to FIG. 6, the process 600 may involve a content source comprising a document containing multiple information modalities (e.g., text, image, video, audio, etc.), where the document is split or parsed into different portions based on the modality of each portion. Then, for all portions represented by non-textual modalities, such as images, textualization may be performed, e.g., textual descriptions (e.g., using MLLMs or based on accompanying descriptions of such objects in the document such as captions that come naturally with images) may be generated. The textual information, including the original text in the document and texts obtained by textualization of the not-textual modalities, may be combined and used to build an initial KG. The KG is further augmented with reasoning (e.g., according to FIG. 5) as well as additional relevant KGs. In some embodiments, when augmenting a KG, an external KG can help expand the KG based on the document with broader and richer information from the digital world. One KG augmentation example is to incorporate information from cited articles in a document into its own KG, thereby providing richer details when generating input for assisting a generative Al system.

[0078] FIG. 7 is a block diagram illustrating aspects associated with obtaining user interaction with a content source in accordance with some embodiments. Various operations or aspects referred to in FIG. 7 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations or aspects referred to in FIG. 7, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0079] As depicted in FIG. 7, a content source (e.g., a document or web page) can be provided to the user, and the user’s interaction with the content can be monitored or captured in various ways. For example, as information is provided to a user, the personalizer system 102 or an entity thereof can track what, where, and how the user focuses on the content, and how the user reacts (e.g., physically or emotionally) to different parts of the content. Example user interaction capture methods may include tracking eye gaze patterns, vocal responses, user scrolling patterns, signs of frustration and / or confusions (e.g., by tracking facial expressions, emotions, or other signs), etc.

[0080] FIG. 8 is a block diagram illustrating aspects associated with building a UCIDS from obtained user interaction information (e.g., as captured in FIG. 7) in accordance with some embodiments. Various operations or aspects referred to in FIG. 8 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, apersonalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations or aspects referred to in FIG. 8, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0081] As depicted in FIG. 8, the shaded regions of a shown document indicate regions of struggles (RoS), where color intensity for a given region represents a corresponding DoS for the user. For example, a higher color intensity (e.g., dark grey) may indicate a higher degree of struggle with the concept(s) contained by a region, while a lower color intensity (e.g., light gray) may indicate a lower degree of struggle (DoS) with the concept(s) contained by a region. The arrows represent the user’s navigations between regions. The thickness of each arrow is proportional to the intensity of user’s navigation from a region to another, as a proxy indicator of potential difficulty that user is experiencing in relating concepts with each other. Note that in FIG. 8 the color intensity and arrow thickness are for illustrative purpose only. In practice, the DoSs may be quantified (e.g., a numeric value or objective measure) for generative Al system(s) or other aspects. In some embodiments, the DoS of a node in UCIDS is determined according to the relative duration of time user is fixed on the region corresponding to the node (and the DoS may also be assigned to all concepts contained in that region). In some embodiments, the centrality of a node determines DoS, e.g., the nodes having high inflow from other nodes are assigned with high DoS. In some embodiments, the DoS of edges in the UCIDS is determined using the frequency of user’s gaze transitions from source node of the edge to the destination node.

[0082] In some embodiments, the personalizer system 102 or one or more entities therein may generate or provide input (e.g., an input prompt or context) for one or more generative Al system(s) (e.g., an assistant LLM). In such embodiments, the input may be for triggering the one or more generative Al system(s) to provide the user with useful responses (e.g., responses that helps to reduce the user’s struggle with grasping particular content). For example, various information may be provided in the input including, for example, textualized UCIDS or portions thereof and textualized subgraph(s) extracted by mapping UCIDS to KG or portions thereof. In this example, Textualized UCIDS data may inform a generative Al system which concepts (and concepts relationships) to focus on and how much the user struggles with those concepts (and concepts relationships) (e.g., as quantified by DoS values) and textualized subgraph(s) may provide information or details from a relevant KG, thereby providing broader context for the generative Al system to use when generating its response.

[0083] FIG. 9 is a flow chart illustrating example operations 900 for generating content assistance in accordance with some embodiments. Various operations from the flow chart of FIG.9 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations of FIG. 9, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0084] Referring to FIG. 9, at block 901, a UCIG or other UCIDS may be created after identifying one or more estimated RoS(s) via user tracking or monitoring (e.g., as indicated in FIG. 8 or via one or more methods or techniques described herein).

[0085] At block 902, the UCIG or other UCIDS may be textualized for further processing (e.g., via the process further described in FIG. 10 or a similar process). For example, UCIG textualization may involve describing in words what concepts the user is struggling to comprehend and may also quantify the associated struggle (e.g., a DoS value may represent as a percentage from 0% to 100% where 100% indicates the highest degree of struggle).

[0086] At block 903, the UCIG or other UCIDS or related data may be mapped to a KG (e.g., an augmented KG generated using the process described in FIG. 7) and may be based on data from a content source, graph reasoning, and / or additional KG(s)). In some embodiments, mapping (e.g., as indicated by the process described in FIG. 11 or a similar process) may include adding or modifying DoS values associated with subgraphs of the KG based on RoS and DoS information indicated by the UCIG or other UCIDS and / or associating subgraphs in the KG with particular RoS(s) indicated by the UCIG or other UCIDS.

[0087] At block 903, relevant subgraphs of the KG (e.g., those that are related to RoS(s) indicated by the UCIG or other UCIDS) may be extracted and textualized. For example, subgraph textualization may involve describing in words what concepts are in a subgraph and may also quantify any DoS associated with each concept therein.

[0088] At block 904, input (e.g., context or an input prompt) for one or more generative Al system(s) may be generated. In some embodiments, generating the input may include using user preferences, a UCIDS, the original content source, and / or output or portions thereof from blocks 901, 902, and / or 903. In some embodiments, the input or subsequent input(s) may be constructed to trigger a generative Al system to generate clarifying explanations or questions to assist in resolving a user’s struggle to comprehend content associated with a user’s RoS(s) and / or to optionally show support or grounding based on the reference materials (e.g., the original content source). In some embodiments, the input or subsequent input(s) may direct the answer or response format from a generative Al system based on content modality preferences.

[0089] In some embodiments, the personalizer system 102 or an entity therein (e.g., a personalizer module illustrated in the bottom right of FIG. 9) may receive user preferences and a UCIDS (or a textualized version) and use this information to provide personalized content assistance via detailed input prompts for one or more generative Al system(s). For example, the personalizer system 102 or an entity therein may provide suggestion, via input prompt for an LLM, on how to present results to a user, e.g., the input prompt may indicate which modalities are preferable for expressing extracted knowledge from the LLM to the user, and may also indicate alternative modalities to use to show content or knowledge associated with RoSs identified in an original content source viewed or interacted with by the user.

[0090] FIG. 10 is a flow chart illustrating an example procedure 1000 for textualizing a UCIDS and extracting concepts and concept relations therefrom in accordance with some embodiments. Various operations in the process of FIG. 10 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations in the process of FIG. 10, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations. For example, the procedure 1000 may be similar to or different from the UCIG textualization referred to in block 902 of FIG. 9. In some embodiments, to obtain a textualized UCIDS when the UCIDS is a spatial graph representing RoSs, the UCIDS may be transformed into a semantic domain in order to textualize the information therein and / or map it to a relevant KG.

[0091] Referring to FIG. 10, the RoSs of a content source (e.g., as indicated by a UCIDS) may be fed to suitable models for analysis, such as object detection (e.g., for visual regions) and optical character recognition (OCR) (e.g., for textual regions). Then, the detected non-textual portions may be described in textual format using one or more approaches, e.g., image to text models (such as CLIP or OpenCLIP). Then, the textual information may be combined and / or analyzed, and the involved concepts (and concepts relationships) may be extracted.

[0092] In some embodiments, extracted concepts (e.g., semantics) derived from the UCIDS may be used in various ways. One way that UCIDS-extracted concepts can be used involves guiding what concepts a generative Al system (e.g., LLM) is to focus on when providing a response to user. This guidance is facilitated because each concept (and concept relationships) (e.g., in a textualized UCIDS or a mapped KG) may be associated with a value representing the DoS (e.g., 90% indicates more struggle (e.g., see block 902 of FIG. 9). Another way that UCIDS- extracted concepts can be used involves extracting relevant subgraphs from a KG to provide asadditional context for a generative Al system to use when generating its response. Identifying or extracting subgraph(s) of a KG is further discussed in various embodiments in the present disclosure, e.g., block 903 of FIG. 9 and in FIG. 11.

[0093] FIG. 11 is a flow chart illustrating an example procedure 1100 for mapping a UCIDS to a KG in accordance with some embodiments. Various operations in the process of FIG. 11 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., anode, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations in the process of FIG. 11, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0094] Referring to FIG. 11, to map an UCIDS to a KG based on a content source, queries may be based on extracted or derived concepts from the UCIDS, and these queries may be applied to the KG. Each query may result in several subgraphs of the KG, where subgraph may have a different diameter (e.g., indicated as maximum number of hops, h, which needs to be traversed between two nodes of the subgraph). The extracted subgraphs, after being textualized (e.g., like in block 903 in FIG. 9) may be included or integrated into an input prompt or context for one or more generative Al system(s). In some embodiments, the personalizer system 102 or an entity thereof may limit the size of extracted subgraphs by indicating a pre-determined hop threshold. For example, the personalizer system 102 or an entity thereof may use a hop threshold in an effort to the input prompt length or size (e.g., so as to not exceed a maximum prompt window size of a particular generative Al system). In this example, if a generative Al system supports a small prompt window size (also called context window), the hop threshold may be set to small values, otherwise large hops may be used which can lead to better responses from generative Al system. In some embodiments, the personalizer system 102 or an entity thereof may start with a small hop threshold and gradually increases the hop threshold, e.g., based on user satisfaction with the answers provided by a generative Al system. In such embodiments, a small hop threshold that leads to user satisfaction may be desirable because, in addition to user satisfaction, this saves the number of computations (since the number of computations generally scales with the context window size).

[0095] FIG. 12 is a flow chart illustrating an example procedure 1200 for adapting a content modality preference and creating a response based on the content modality preference in accordance with some embodiments. Various operations in the process of FIG. 12 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, thepersonalizer system 102 or one or more entities therein may be configured for performing one or more operations in the process of FIG. 12, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0096] In some embodiments, the personalizer system 102 or one or more entities therein may instruct or direct a generative Al system (e.g., an assistant MLLM) to generate and present answers or responses (potentially, including additional content not in an original content source) using content modalities preferred by the user. In some embodiments, the personalizer system 102 or one or more entities therein may instruct or direct a generative Al system to provide references to portions in the original document as supporting evidence when generating a response or answer. In some embodiments, the generative Al system may optionally follow the instructions in the prompt, e.g., a MLLM may transform or provide a response or portions thereof corresponding to RoS(s) of a user in a preferred content modality (e.g., images, tables, and text instead of only text).

[0097] FIG. 13 is a flow chart illustrating an example refinement procedure 1300 for refining content modality preferences in accordance with some embodiments. In some embodiments, the refinement procedure may observe and analyze a user’s experience (e.g., with a response from a generated Al system) and update one or more content modality preference based on this information. Various operations in the process of FIG. 13 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations in the process of FIG. 13, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0098] In some embodiments, a user may have an initial preference of what modality to use in multi-modal LLM response to present different concepts included in the response. For example, a user may prefer text modality for math concepts while they prefer vision modality for art related concepts. Some concepts might also be preferred to be presented in multiple modalities such as presenting chemical processes in text as well as vision. The user can indicate an initial preference of modalities for a coarse-grained set of concepts like, {math, art, chemistry, ... }, but can refine it to a fine-grained preference set per individual concepts (or concept relationships). This can be learned in an iterative fashion over a course of user interactions with the system.

[0099] Referring to FIG. 13, the refinement procedure 1300 for refining modality preferences may include: providing an input prompt to an LLM or other generative Al system (e.g., accordingly to block 904 in FIG. 9; obtaining a response ‘R’ from the LLM or other generative Al system; parsing all concepts from the response ‘R’; choosing a concept ‘c’ until all conceptsin the response ‘R’ are evaluated. The refinement procedure 1300 may also include, for each concept ‘c’ in the response ‘R’, updating or adding the concept ‘c’ to a modality ‘m’ in a coarsegrained modality preference (CGM) dataset; rendering or presenting the concept ‘c’ to the user using the modality ‘m’, measuring the user experience and / or feedback, if the user is satisfied with the rendering or presentation using the modality ‘m’, adding or updating an association between the concept ‘c’ and the modality ‘m’ in a modified modality preference (MMP) dataset, and if no concepts are left to be evaluated in the response ‘R’, ending the refinement procedure 1300 and otherwise choosing another concept to evaluate and performing similar operations.

[0100] The refinement process may also include: if the user is not satisfied with the render or presentation using the modality ‘m’, determining whether all candidate modalities ‘m’ in a feasible modalities per concept (FMC) dataset have been evaluated and, if at least one candidate modality ‘m’ is unevaluated, choosing an unevaluated candidate modality ‘m’ and rendering or presenting the concept ‘c’ to the user using the candidate modality ‘m’, measuring the user experience and / or feedback, if the user is satisfied with the rendering or presentation using the candidate modality ‘m’, adding or updating an association between the concept ‘c’ and the candidate modality ‘m’ in the MMP dataset and if no concepts are left to be evaluated in the response ‘R’, ending the refinement procedure 1300 and otherwise choosing another concept from the response ‘R’ to evaluate and performing similar operations.

[0101] The refinement process may also include: if the user is not satisfied with the render or presentation using the candidate modality ‘m’, determining that at least one candidate modality ‘m’ is unevaluated, choosing a next unevaluated candidate modality ‘m’ and rendering or presenting the concept ‘c’ to the user using the candidate modality ‘m’, measuring the user experience and / or feedback, if the user is not satisfied with the rendering or presentation using the candidate modality ‘m’, choosing a next unevaluated candidate modality ‘m’ and evaluating it using similar operations until the user is satisfied with the current candidate modality or until all candidate modalities in the FMC data are evaluated.

[0102] The refinement process may also include: when all candidate modalities in the FMC data have been evaluated for a concept ‘c’, adding or updating an association between the concept ‘c’ and a fall back or default modality ‘m’ in the CGM dataset and if no concepts are left to be evaluated in the response ‘R’, ending the refinement procedure 1300 and otherwise choosing another concept from the response ‘R’ to evaluate and performing similar operations.

[0103] In some embodiments, the personalizer system 102 or one or more entities therein may be configured to prioritize or check whether a particular dataset has relevant content modality preferences in a particular order. For example, the personalizer system 102 or one or more entitiestherein may utilize a relevant content modality preference from a MMP dataset if one exists before using a content modality preference from a CGM dataset or FMC dataset.

[0104] FIG. 14 is a flow chart illustrating examples of operations for providing or generating personalized content assistance in accordance with some embodiments. Various operations from the flow chart of FIG. 14 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations of FIG. 14, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0105] In some embodiments, the personalizer system 102 or one or more entities therein (e.g., implemented using the structure of FIG. 18, 19, or 20) will now be discussed with reference to the flow chart of FIG. 14 according to some embodiments of the present disclosure. For example, modules (e.g., a personalizer module or an input generator) may be stored in memory 1810 of FIG. 18, and these modules may provide instructions so that when the instructions of a module are executed by respective processing circuitry 1802, device 1800 performs respective operations of the flow chart of FIG. 14. In another example, one or more modules or functionality may be distributed among multiple devices, platforms, or components therein.

[0106] Referring to FIG. 14, at block 1400, a first UCIDS is created using user interaction information associated with a user interacting with a first content source (e.g., a data file, a website, a web document, etc.) wherein the first UCIDS indicates at least one estimated RoS for the user. For example, the UCIDS may be in the form of a spatial graph comprising nodes representing different portions of a document; edges representing transitions or interactions between portions; and labels representing various information, e.g., DoS values, attributes, etc.

[0107] At block 1402, the first UCIDS is mapped to a KG based on the first content source. For example, mapping a first UCIDS to a KG may include converting the UCIDS from a spatial graph to a semantic graph, and then identifying subgraphs in the KG that are related to RoS(s) indicated by the first UCIDS. In some embodiments, mapping may also include adding or modifying DoS values (e.g., where original DoS values are from a first UCIDS) associated with subgraphs of the KG based on information from various sources, e.g., a subsequent UCIDS, HRoS, user feedback, etc.

[0108] In some embodiments, a KG is generated using textualized content data and, optionally, other information.

[0109] In some embodiments, a KG is augmented using inferred facts based on graph reasoning and / or one or more additional knowledge graphs (e.g., broader KGs, public KGs, external KGs, etc.).

[0110] At block 1404, at least one subgraph of the KG associated with the least one estimated RoSs is determined using the mapping. For example, after mapping a first UCIDS to a KG, when creating an input prompt, at least one subgraph of the KG associated with a user’s estimated RoSs may be utilized to trigger a generative Al system to generate clarifying explanations or questions to assist in resolving a user’s struggle to comprehend content associated with a user’s RoS(s).

[0111] At block 1406, a first input for one or more generative Al system(s) is generated using the at least one subgraph of the knowledge graph. For example, the personalizer system 102 or an entity therein may generate an input prompt based on data obtained from a subgraph of a KG associated with a user’s RoS(s).

[0112] At block 1408, the first input may be sent to the one or more generative Al system(s). For example, the personalizer system 102 or an entity therein may send a personalized input prompt to an LLM, MLLM, or another Al system. In this example, the personalized input prompt may be created or designed to obtain content (e.g., especially content that the user struggles to understand) in a format or modality that makes it easier to consume or comprehend.

[0113] In some embodiments, generating a first input includes applying a heuristic-based method to control the length of the first input or a portion thereof such that the first input does not exceed a permitted context length associated with the one or more generative Al system(s).

[0114] In some embodiments, generating a first input includes generating an input prompt comprising textualized content data, source content data, textualized RoS data, and / or textualized subgraph data.

[0115] In some embodiments, generating an input prompt includes specifying a response format or modality for one or more concepts and / or providing instructions to identify portions of the first content source as supporting evidence when responding.

[0116] In some embodiments, at least some user interaction information is obtained by monitoring the user while the user is interacting with the content source.

[0117] In some embodiments, monitoring a user includes eye tracking; head tracking; gaze tracking; emotion recognition analysis; behavioral analysis; video analysis; sound analysis; user reaction analysis; biosensor(s) analysis; and / or user feedback analysis.

[0118] In some embodiments, at least some user interaction information includes user settings, user profile information, information from prior user interactions, or interaction information from at least one other user.

[0119] In some embodiments, a content source includes a text document; a document with multi-modal data; multimedia content; a web page or site; a web page or site with text, images, video, and / or sound; or a data file comprising text, images, video, and / or sound.

[0120] In some embodiments, a UCIDS includes one or more lists, a data tree, or a spatial graph.

[0121] In some embodiments, one or more generative Al system(s) includes a large language model (LLM), a multi-modal LLM, a multi-modal model, a large multi-modal model, or a generative adversarial network (GAN).

[0122] In some embodiments, the method or operation(s) of FIG. 14 is performed by a system including a device, a node, a computer, a user device, a network node, a web server, or a distributed computer system.

[0123] FIG. 15 is a flow chart illustrating examples of operations to adjust or modify user experience and / or user preference(s) associated with generating personalized content assistance in accordance with some embodiments. Various operations from the flow chart of FIG. 15 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., anode, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations of FIG. 15, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0124] In some embodiments, the personalizer system 102 or one or more entities therein (e.g., implemented using the structure of FIG. 18, 19, or 20) will now be discussed with reference to the flow chart of FIG. 15 according to some embodiments ofthe present disclosure. For example, modules (e.g., a personalizer module or an input generator) may be stored in memory 1810 of FIG. 18, and these modules may provide instructions so that when the instructions of a module are executed by respective processing circuitry 1802, device 1800 performs respective operations of the flow chart of FIG. 15. In another example, one or more modules or functionality may be distributed among multiple devices, platforms, or components therein.

[0125] In some embodiments, the personalizer system 102 or an entity thereof may determine adjust or modify how content is personalized based on user feedback and / or other information. For example, the personalizer system 102 or an entity thereof may add or adjust a user profile or related user preferences such that personalized content assistance can dynamically change as the user or similar users consume various content and provides related feedback.

[0126] Referring to FIG. 15, at block 1500, feedback is received associated with a first input. In some embodiments, the feedback may include a response from the one or more generative Alsystem(s), user input associated with the first input or the response, or user reaction to the first input or the response.

[0127] At block 1502, user experience and / or user preference(s) associated with generating personalized content assistance is adjusted using the feedback. In some embodiments, adjusting user experience and / or user preference(s) includes creating or modifying a modality preference for presenting a concept or relationship(s) between concepts, wherein the modality preference indicates a preferred format or modality for receiving information associated with the concept.

[0128] In some embodiments, adjusting user experience and / or user preference(s) includes adding or removing an estimated RoS or a related concept for the user or modifying an estimated DoS value associated with an RoS or a related concept for the user.

[0129] In some embodiments, adjusting user experience and / or user preference(s) includes generating an additional input prompt to clarify the first input or the response.

[0130] FIG. 16 is a flow chart illustrating examples of operations to add or modify a content modality preference in accordance with some embodiments. Various operations from the flow chart of FIG. 16 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., anode, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations of FIG. 16, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0131] In some embodiments, the personalizer system 102 or one or more entities therein (e.g., implemented using the structure of FIG. 18, 19, or 20) will now be discussed with reference to the flow chart of FIG. 16 according to some embodiments ofthe present disclosure. For example, modules (e.g., a personalizer module or an input generator) may be stored in memory 1810 of FIG. 18, and these modules may provide instructions so that when the instructions of a module are executed by respective processing circuitry 1802, device 1800 performs respective operations of the flow chart of FIG. 16. In another example, one or more modules or functionality may be distributed among multiple devices, platforms, or components therein.

[0132] In some embodiments, the personalizer system 102 or an entity thereof may determine how content received from a generative Al system is to be expressed or formatted and may provide or facilitate providing the content using this modality. In some embodiments, the personalizer system 102 or an entity thereof may utilize one or more types of content modality preferences, where some content modalities preferences are preconfigured and / or refined (e.g., via a refinement procedure similar to or the same as that depicted in FIG. 13).

[0133] Referring to FIG. 16, at block 1600, a response is received from one or more generative Al system(s). For example, the personalizer system 102 or an entity thereof may receive a response from an LLM that contains information about various concepts or topics including concepts that a user may have trouble comprehending.

[0134] At block 1602, the response is parsed or analyzed to obtain or identify one or more concepts to present to the user. For example, the personalizer system 102 or an entity thereof may parse a response to identify concepts expressed in the response.

[0135] At block 1604, for each of the concepts that lacks a modality preference in a userspecific modality present data set: an iterative process may be performed. For example, the iterative process may comprise: selecting, from a predefined set of associations between concepts and modality preferences, a modality preference for presenting the concept; presenting the concept based on the modality preference; determining, based on user feedback, whether the user is satisfied with the presentation of the concept; if the user is deemed satisfied with the presentation of the concept, adding the modality preference to the user-specific modality present data set; and if the user is deemed unsatisfied with the presentation of the concept, presenting the concept using a different candidate modality preference and determining user satisfaction with the new presentation until the user is deemed satisfied and the candidate modality preference is added to the user-specific modality present data set or until all candidate modality preferences have been evaluated or utilized.

[0136] FIG. 17 is a flow chart illustrating examples of operations to map a second UCIDS to an existing KG in accordance with some embodiments. Various operations from the flow chart of FIG. 17 may be optional with respect to some embodiments of the personalizer system 102, an entity thereof (e.g., a node, a device, a personalizer module, etc.), and / or related methods. In some embodiments, the personalizer system 102 or one or more entities therein may be configured for performing one or more operations of FIG. 17, e.g., using a processing circuitry and / or a memory comprising instructions for performing such operations.

[0137] In some embodiments, the personalizer system 102 or one or more entities therein (e.g., implemented using the structure of FIG. 18, 19, or 20) will now be discussed with reference to the flow chart of FIG. 17 according to some embodiments of the present disclosure. For example, modules (e.g., a personalizer module or an input generator) may be stored in memory 1810 of FIG. 18, and these modules may provide instructions so that when the instructions of a module are executed by respective processing circuitry 1802, device 1800 performs respective operations of the flow chart of FIG. 17. In another example, one or more modules or functionality may be distributed among multiple devices, platforms, or components therein.

[0138] Referring to FIG. 17, at block 1700, a second UCIDS is created using second user interaction information associated with a user interacting with a second content source. Creating the second UCIDS includes using data from a first UCIDS or historical RoS data. For example, historical RoS data may include RoS data derived from a user interacting with a first content source or similar content, RoS data derived from the user interacting with other content, and / or RoS data derived from other users interacting with the same or similar content.

[0139] At block 1702, the second UCIDS is mapped to a KG. For example, mapping a second UCIDS to a KG may include converting the UCIDS from a spatial graph to a semantic graph, and then identifying subgraphs in the KG that are related to RoS(s) indicated by the second UCIDS. In this example, mapping may also include adding or modifying DoS values associated with subgraphs of the KG based on information in the second UCIDS.

[0140] At block 1704, it is determined, using the mapping of the second UCIDS to the knowledge graph, one or more subgraph(s) of the KG associated with a user’s estimated RoSs. For example, prior to a second UCIDS being generated, when creating an input prompt, at least one subgraph of the KG associated with a user’s estimated RoSs may be utilized to trigger a generative Al system to generate clarifying explanations or questions to assist in resolving a user’s struggle to comprehend content associated with a user’s RoS(s). In this example, after the second UCIDS is mapped to the KG, when creating a subsequent input prompt, the existing subgraph or a different subgraph of the KG associated with a user’s estimated RoSs may be utilized.

[0141] FIG. 18 shows a device 1800 in accordance with some embodiments. In some embodiments, device 1800 may include a UE or a device capable, configured, arranged and / or operable to communicate wirelessly with nodes and / or other devices. Examples of a device include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage / playback device, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptopmounted equipment (LME), an Augmented Reality (AR) or Virtual Reality (VR) device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples of device 1800 may include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band (NB) internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0142] A device may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a device or UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a device or UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a device or UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0143] The device 1800 includes processing circuitry 1802 that is operatively coupled via a bus 1804 to an input / output interface 1806, a power source 1808, a memory 1810, a communication interface 1812, and / or any other component, or any combination thereof. Certain devices may utilize all or a subset of the components shown in FIG. 18. The level of integration between the components may vary from one device to another device. Further, certain devices may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0144] The processing circuitry 1802 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 1810. The processing circuitry 1802 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 1802 may include multiple central processing units (CPUs).

[0145] In the example, the input / output interface 1806 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the device 1800. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. Anoutput device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0146] In some embodiments, the power source 1808 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 1808 may further include power circuitry for delivering power from the power source 1808 itself, and / or an external power source, to the various parts of the device 1800 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 1808. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 1808 to make the power suitable for the respective components of the device 1800 to which power is supplied.

[0147] The memory 1810 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 1810 includes one or more application programs 1814, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 1816. The memory 1810 may store, for use by the device 1800, any of a variety of various operating systems or combinations of operating systems.

[0148] The memory 1810 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a universal mobile telecommunications system (UMTS) SIM (USIM) and / or an integrated SIM (ISIM), other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 1810 may allow the device 1800 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 1810, which may be or comprise a device-readable storage medium.

[0149] The processing circuitry 1802 may be configured to communicate with an access network or other network using the communication interface 1812. The communication interface 1812 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 1822. The communication interface 1812 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another device or a node in a computer network). Each transceiver may include a transmitter 1818 and / or a receiver 1820 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 1818 and receiver 1820 may be coupled to one or more antennas (e.g., antenna 1822) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0150] In the illustrated embodiment, communication functions of the communication interface 1812 may include cellular communication, Wi-Fi communication, low-powered wired area network (LPWAN) communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Global System for Communications (GSM), Long Term Evolution (LTE), New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / intemet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), quick UDP Internet connections (QUIC), Hypertext Transfer Protocol (HTTP), and so forth.

[0151] Regardless of the type of sensor, a device or UE may provide an output of data captured by its sensors, through its communication interface 1812, via a wireless connection to a network node. Data captured by sensors of a device or UE can be communicated through a wireless connection to a node via another device or UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0152] As another example, a device comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wirelessconnection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the device may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0153] A device or UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A device or UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the device 1800 shown in FIG. 18.

[0154] As yet another specific example, in an loT scenario, a device or UE may represent a machine, a node, or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another device and / or a node. The device or UE may in this case be a machine-to-machine (M2M jdevice, which may in a 3 GPP context be referred to as an MTC device. As one particular example, the device or UE may implement the 3GPP NB-IoT standard. In other scenarios, a device or UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0155] In practice, any number of devices may be used together with respect to a single use case. For example, a first device or UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second device or UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first device or UE may adjust the throttle on the drone (e.g., by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second device or UE can also include more thanone of the functionalities described above. For example, a device or UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0156] FIG. 19 shows a node 1900 in accordance with some embodiments. In some embodiments, node 1900 may represent a network node or a computing device capable of communicating via a network (e.g., a telecommunications network, a communications network, or an Internet network). For example, a node may refer to equipment capable, configured, arranged, and / or operable to communicate directly or indirectly with a device and / or with other nodes or equipment in a telecommunication network or another network. Examples of nodes include, but are not limited to, servers, hosts, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU).

[0157] Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0158] Other examples of nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0159] The node 1900 includes a processing circuitry 1902, a memory 1904, a communication interface 1906, and a power source 1908. The node 1900 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the node 1900 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, eachunique NodeB and RNC pair, may in some instances be considered a single separate node. In some embodiments, the node 1900 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 1904 for different RATs) and some components may be reused (e.g., a same antenna 1910 may be shared by different RATs). The node 1900 may also include multiple sets of the various illustrated components for different wireless technologies integrated into node 1900, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within node 1900.

[0160] The processing circuitry 1902 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other node 1900 components, such as the memory 1904, to provide node 1900 functionality.

[0161] In some embodiments, the processing circuitry 1902 includes a system on a chip (SOC). In some embodiments, the processing circuitry 1902 includes one or more of radio frequency (RF) transceiver circuitry 1912 and baseband processing circuitry 1914. In some embodiments, the radio frequency (RF) transceiver circuitry 1912 and the baseband processing circuitry 1914 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 1912 and baseband processing circuitry 1914 may be on the same chip or set of chips, boards, or units.

[0162] The memory 1904 may comprise any form of volatile or non-volatile computer- readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 1902. The memory 1904 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 1902 and utilized by the node 1900. The memory 1904 may be used to store any calculations made by theprocessing circuitry 1902 and / or any data received via the communication interface 1906. In some embodiments, the processing circuitry 1902 and memory 1904 are integrated.

[0163] The communication interface 1906 is used in wired or wireless communication of signaling and / or data between a node, a device, an access network, and / or a UE. As illustrated, the communication interface 1906 comprises port(s) / terminal(s) 1916 to send and receive data, for example to and from a network over a wired connection. The communication interface 1906 also includes radio front-end circuitry 1918 that may be coupled to, or in certain embodiments a part of, the antenna 1910. Radio front-end circuitry 1918 comprises filters 1920 and amplifiers 1922. The radio front-end circuitry 1918 may be connected to an antenna 1910 and processing circuitry 1902. The radio front-end circuitry may be configured to condition signals communicated between antenna 1910 and processing circuitry 1902. The radio front-end circuitry 1918 may receive digital data that is to be sent out to other nodes or devices via a wireless connection. The radio front-end circuitry 1918 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 1920 and / or amplifiers 1922. The radio signal may then be transmitted via the antenna 1910. Similarly, when receiving data, the antenna 1910 may collect radio signals which are then converted into digital data by the radio front-end circuitry 1918. The digital data may be passed to the processing circuitry 1902. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0164] In certain alternative embodiments, the node 1900 does not include separate radio front-end circuitry 1918, instead, the processing circuitry 1902 includes radio front-end circuitry and is connected to the antenna 1910. Similarly, in some embodiments, all or some of the RF transceiver circuitry 1912 is part of the communication interface 1906. In still other embodiments, the communication interface 1906 includes one or more ports or terminals 1916, the radio frontend circuitry 1918, and the RF transceiver circuitry 1912, as part of a radio unit (not shown), and the communication interface 1906 communicates with the baseband processing circuitry 1914, which is part of a digital unit (not shown).

[0165] The antenna 1910 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 1910 may be coupled to the radio front-end circuitry 1918 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 1910 is separate from the node 1900 and connectable to the node 1900 through an interface or port.

[0166] The antenna 1910, communication interface 1906, and / or the processing circuitry 1902 may be configured to perform any receiving operations and / or certain obtaining operationsdescribed herein as being performed by the node. Any information, data and / or signals may be received from a device, a UE, another node, and / or any other network equipment or computing equipment. Similarly, the antenna 1910, the communication interface 1906, and / or the processing circuitry 1902 may be configured to perform any transmitting operations described herein as being performed by the node. Any information, data and / or signals may be transmitted to a device, a UE, another node, and / or any other network equipment or computing equipment.

[0167] The power source 1908 provides power to the various components of node 1900 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 1908 may further comprise, or be coupled to, power management circuitry to supply the components of the node 1900 with power for performing the functionality described herein. For example, the node 1900 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 1908. As a further example, the power source 1908 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0168] Embodiments of the node 1900 may include additional components beyond those shown in FIG. 19 for providing certain aspects of the node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the node 1900 may include user interface equipment to allow input of information into the node 1900 and to allow output of information from the node 1900. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the node 1900. In some embodiments, some components depicted in FIG. 19, such as the radio front-end circuitry 1918 and the RF transceiver circuitry 1912, may be omitted.

[0169] FIG. 20 is a block diagram illustrating a virtualization environment 2000 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 2000 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a node, UE, a network node, a core network node, or a host. Further, inembodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 2000 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an 0-2 interface. Virtualization may facilitate distributed implementations of a network node, UE, core network node, or host.

[0170] Application(s) 2002 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein.

[0171] Hardware 2004 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 2006 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 2008a and 2008b (one or more of which may be generally referred to as VMs 2008), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 2006 may present a virtual operating platform that appears like networking hardware to the VMs 2008.

[0172] The VMs 2008 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 2006. Different embodiments of the instance of a virtual appliance 2002 may be implemented on one or more of VMs 2008, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0173] In the context of NFV, a VM 2008 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 2008, and that part of hardware 2004 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 2008 on top of the hardware 2004 and corresponds to the application 2002.

[0174] Hardware 2004 may be implemented in a standalone node (e.g., a network node) with generic or specific components. Hardware 2004 may implement some functions via virtualization. Alternatively, hardware 2004 may be part of a larger cluster of hardware (e.g., such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 2010, which, among others, oversees lifecycle management of applications 2002. In some embodiments, hardware 2004 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 2012 which may alternatively be used for communication between hardware nodes and radio units.

[0175] Although the computing devices described herein (e.g., UEs, devices, network nodes, hosts, etc.) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non-computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0176] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

Claims

CLAIMSWhat is Claimed is:

1. A method for generating personalized content assistance using generative artificial intelligence (Al), the method comprising: creating (1400) a first user-content interaction data structure (UCIDS) using user interaction information associated with a user interacting with a first content source, wherein the first UCIDS indicates at least one estimated region of struggle (RoS) for the user; mapping (1402) the first UCIDS to a knowledge graph based on the first content source; determining (1404), using the mapping, at least one subgraph of the knowledge graph associated with the at least one estimated RoS; generating (1406), using the at least one subgraph of the knowledge graph, a first input for one or more generative Al system(s); and sending (1408) the first input to the one or more generative Al system(s).

2. The method of Claim 1 , comprising: receiving (1500) feedback associated with the first input; and adjusting (1502), using the feedback, user experience and / or user preference(s) associated with providing personalized content assistance.

3. The method of Claim 2, wherein the feedback includes a response from the one or more generative Al system(s), user input associated with the first input or the response, or user reaction to the first input or the response.

4. The method of any one of Claims 2 and 3, wherein adjusting (1502) user experience and / or user preference(s) includes creating or modifying a modality preference for presenting a concept or relationship(s) between concepts, wherein the modality preference indicates a preferred format or modality for receiving information associated with the concept.

5. The method of any one of Claims 2 to 4, wherein adjusting (1502) user experience and / or user preference(s) includes adding or removing an estimated RoS or a related concept for the user or modifying an estimated degree of struggle (DoS) value associated with an RoS or a related concept for the user.

6. The method of any one of Claims 2 to 5, wherein adjusting (1502) user experience and / or user preference(s) includes generating an additional input prompt to clarify the first input or the response.

7. The method of Claim 1 , comprising: receiving (1600) a response from the one or more generative Al system(s); parsing (1602) the response to obtain one or more concepts to present to the user; for each of the concepts that lacks a modality preference in a user-specific modality present data set (1604): selecting, from a predefined set of associations between concepts and modality preferences, a modality preference for presenting the concept; presenting the concept based on the modality preference; determining, based on user feedback, whether the user is satisfied with the presentation of the concept; if the user is deemed satisfied with the presentation of the concept, adding the modality preference to the user-specific modality present data set; and if the user is deemed unsatisfied with the presentation of the concept, presenting the concept using a different candidate modality preference and determining user satisfaction with the presentation associated with the candidate modality preference until the user is deemed satisfied and the candidate modality preference is added to the userspecific modality present data set or until all candidate modality preferences have been evaluated.

8. The method of any one of Claims 1 to 7, comprising: creating (1700) a second UCIDS using second user interaction information associated with a user interacting with a second content source; wherein creating the second UCIDS includes using data from the first UCIDS or historical RoS data; mapping (1702) the second UCIDS to the knowledge graph; and determining (1704), using the mapping of the second UCIDS to the knowledge graph, the at least one subgraph or a different subgraph of the knowledge graph.

9. The method of Claim 8, wherein the historical RoS data includes RoS data derived from the user interacting with the first content source or similar content, RoS data derived fromthe user interacting with other content, and / or RoS data derived from other users interacting with the same content or similar content.

10. The method of any one of Claims 1 to 9, wherein the knowledge graph is generated using textualized content data.

11. The method of any one of Claims 1 to 10, wherein the knowledge graph is augmented using inferred facts based on graph reasoning and / or one or more additional knowledge graphs.

12. The method of any one of Claims 1 to 11, wherein generating (1406) the first input includes applying a heuristic-based method to control the length of the first input or a portion thereof such that the first input does not exceed a permitted context length associated with the one or more generative Al system(s).

13. The method of any one of Claims 1 to 12, wherein generating (1406) the first input includes generating an input prompt comprising textualized content data, source content data, textualized RoS data, and / or textualized subgraph data.

14. The method of Claim 13, wherein generating the first input prompt includes specifying a response format or modality for one or more concepts and / or providing instructions to identify portions of the first content source as supporting evidence when responding.

15. The method of any one of Claims 1 to 14, wherein at least some of the user interaction information is obtained by monitoring the user while the user is interacting with the first content source.

16. The method of Claim 15, wherein monitoring the user includes eye tracking; head tracking; gaze tracking; emotion recognition analysis; behavioral analysis; video analysis; sound analysis; user reaction analysis; biosensor(s) analysis; and / or user feedback analysis.

17. The method of any one of Claims 1 to 16, wherein at least some of the user interaction information includes user settings, user profile information, information from prior user interactions, or interaction information from at least one other user.

18. The method of any one of Claims 1 to 17, wherein the content source includes a text document; a document with multi-modal data; multimedia content; a web page or site; a web page or site with text, images, video, and / or sound; or a data file comprising text, images, video, and / or sound.

19. The method of any one of Claims 1 to 18, wherein the UCIDS includes one or more lists, a data tree, or a spatial graph.

20. The method of any one of Claims 1 to 19, wherein the one or more generative Al system(s) includes a large language model (LLM), a multi-modal LLM, a multi-modal model, a large multi-modal model, or a generative adversarial network (GAN).

21. The method of any one of Claims 1 to 20, wherein the method or operation(s) thereof is performed by a system including a device, a node, a computer, a user device, a network node, a web server, or a distributed computer system.

22. A system (102, 1800, 1900, 2000) for generating personalized content assistance using generative artificial intelligence (Al), the system comprising: processing circuitry (1802, 1902); a memory (1810, 1904) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the system to perform operations comprising: creating (1400) a first user-content interaction data structure (UCIDS) using user interaction information associated with a user interacting with a first content source, wherein the first UCIDS indicates at least one estimated region of struggle (RoS) for the user; mapping (1402) the first UCIDS to a knowledge graph based on the first content source; determining (1404), using the mapping, at least one subgraph of the knowledge graph associated with the at least one estimated RoS; generating (1406), using the at least one subgraph of the knowledge graph, a first input for one or more generative Al system(s); and sending (1408) the first input to the one or more generative Al system(s).

23. The system (102, 1800, 1900, 2000) of Claim 22, wherein the operations further comprise any of the operations of Claims 2 to 20.

24. The system (102, 1800, 1900, 2000) of any one of Claims 22 and 23, wherein the system comprises a device, a node, a computer, a user device, a network node, a web server, or a distributed computer system.

25. A device (1800) for generating personalized content assistance using generative artificial intelligence (Al), the device comprising: processing circuitry (1802); memory (1810) coupled with the processing circuitry, wherein the memory includes instructions that when executed by the processing circuitry causes the device to perform operations comprising: creating (1400) a first user-content interaction data structure (UCIDS) using user interaction information associated with a user interacting with a first content source, wherein the first UCIDS indicates at least one estimated region of struggle (RoS) for the user; mapping (1402) the first UCIDS to a knowledge graph based on the first content source; determining (1404), using the mapping, at least one subgraph of the knowledge graph associated with the at least one estimated RoS; generating (1406), using the at least one subgraph of the knowledge graph, a first input for one or more generative Al system(s); and sending (1408) the first input to the one or more generative Al system(s).

26. The device (1800) of Claim 25, wherein the operations further comprise any of the operations of Claims 2 to 20.

27. A non-transitory computer readable medium (1810, 1904) including program code (1814) to be executed by processing circuitry (1802, 1902) of a system (102, 1800, 1900, 2000) for generating personalized content assistance using generative artificial intelligence (Al), whereby execution of the program code causes the program code to perform operations comprising: creating (1400) a first user-content interaction data structure (UCIDS) using user interaction information associated with a user interacting with a first content source, wherein the first UCIDS indicates at least one estimated region of struggle (RoS) for the user;mapping (1402) the first UCIDS to a knowledge graph based on the first content source; determining (1404), using the mapping, at least one subgraph of the knowledge graph associated with the at least one estimated RoS; generating (1406), using the at least one subgraph of the knowledge graph, a first input for one or more generative Al system(s); and sending (1408) the first input to the one or more generative Al system(s).

28. The non-transitory computer readable medium (1810, 1904) of Claim 27, the operations further comprising any of the operations of Claims 2 to 20.

Images