Privacy-aware multimodal automated response

Abstract

Various embodiments include systems and methods for generating privacy-aware multimodal auto-replies to incoming communications. A processing system of a computing device may collect multimodal information, determine the current user situation based on the collected information, determine the user's privacy preferences for auto-reply responses, generate prompts to be input to a Generating Large-Scale Language Model (LLM) to generate optional auto-reply responses, receive a list of personal response suggestions from the Generating LLM, and perform auto-reply actions based on the selected personal response suggestions.

Description

Technical Field

[0001] Cross-Reference to Related Applications This application claims the benefit of priority from U.S. Non-Provisional Patent Application No. 18 / 454,456 filed on August 23, 2023, the entire content of which is incorporated herein by reference.

Background Art

[0002] Auto-reply is a function for generating automated actions or responses in a computer system or software application when a specific event or condition (such as receipt of an email or a support request) is satisfied. Auto-reply functions are generally used in text applications, email systems, customer support tools, and social media platforms to confirm receipt or provide preliminary information when a human responder is unavailable to respond. These automated responses may include general information, pre-written answers to frequently asked questions, instructions for further actions, and the like.

[0003] In recent years, auto-reply has become an important function on a variety of different platforms and devices, including conventional mobile operating systems such as Android and iOS (registered trademark). The auto-reply function can assist users in setting up automatic replies for various forms of communication including phone calls, text messages, or emails. The auto-reply function can be particularly useful in situations where a prompt response to incoming communication is not possible, such as during a meeting, while traveling in a vehicle, or in circumstances where the user is in an area without wireless service.

[0004] Messages generated by autoreplies can be pre-configured, customized, or a combination of both, ranging from simple notifications to more detailed explanations. Some systems natively integrate autoreply functionality within their built-in messaging or email settings, while others may rely on third-party apps available from app stores or software repositories. When autoreply is activated, a predefined response may be automatically sent in response to an incoming call or message. Several platforms offer sophisticated configurations that allow for customization of autoreply messages, including specifying specific contacts for autoreplies and defining unique time periods during which autoreplies are active. [Overview of the project]

[0005] Various embodiments include methods that can be implemented in a computing device processing system for providing automated response responses, which may include: collecting multimodal information about the user of the computing device; determining the current user status based on the collected multimodal information; determining the user's privacy preferences for automated response responses; generating prompts based on the selected multimodal information and the user's privacy preferences for automated response responses, and inputting the prompts into a generating large language model (LLM); receiving a list of personal response suggestions from the generating LLM; receiving a user input selection of one of the received personal response suggestions in response to rendering the received personal response suggestions on the computing device's electronic display; and performing an automated response action based on the received user input.

[0006] Some embodiments may further include activating or deactivating one or more information submodules configured to receive data input and output text suitable for use when prompting the generating LLM, based on the determined current user status and the user's privacy preferences for the determined automated response. Some embodiments may further include processing non-text-based information by at least one of the active information submodules to generate text suitable for input to the generating LLM. In some embodiments, generating a prompt may include generating a prompt by combining text-based information and non-text-based information. In some embodiments, non-text-based information includes descriptions based on audio or video sensor data.

[0007] Some embodiments may further include selecting a model size to be used in an information submodule to process non-text-based information based on one or more of the determined current user status, the context of the incoming communication, or the user's privacy preferences regarding automated reply responses.

[0008] In some embodiments, performing an automated reply action based on received user input may include generating a privacy-aware, multimodal generated automated reply message based on the received user input, and sending the generated privacy-aware, multimodal generated automated reply message to the computing device that initiated the incoming call or message. In some embodiments, determining a user's privacy preferences for an automated reply may include determining a user's privacy preferences for an automated reply based on information that identifies categories of information that the user may allow to be included in the automated reply, based on at least one of the context of the incoming call or message, the caller of the incoming call or message, the user's current location, or the user's current activity.

[0009] Some embodiments may further include providing a machine learning module with a user input selection of one of the received personal response suggestions, in order to enable improved prompt generation based on selected multimodal information and the user's privacy preferences for automated response suggestions.

[0010] Further embodiments may include a computing device having a processor comprised of processor-executable instructions for performing various operations corresponding to the method summarized above. Further embodiments may include a non-temporary processor-readable storage medium storing processor-executable instructions configured to cause the processor to perform various operations corresponding to the method summarized above. Further embodiments may include a computing device having various means for performing functions corresponding to the operation of the method summarized above.

[0011] The accompanying drawings, incorporated herein and constituting part thereof, illustrate exemplary embodiments of the claims and, together with the given schematic description and “Modes for Carrying Out the Invention,” are useful in illustrating the features described herein. [Brief explanation of the drawing]

[0012] [Figure 1] This is a component block diagram showing an exemplary computing system that can be configured to implement several embodiments. [Figure 2A] This is a block diagram of components showing the components of a computing system configured to generate privacy-aware, multimodal, and automated responses, according to several embodiments. [Figure 2B] This is a block diagram of components showing the components of a computing system configured to generate privacy-aware, multimodal, and automated responses, according to several embodiments. [Figure 3] This is a process flow diagram illustrating a method for generating privacy-aware, multimodal, automated replies according to several embodiments. [Figure 4] This is a component block diagram showing an exemplary computing device suitable for use with various embodiments. [Figure 5] This is a component block diagram showing an exemplary wireless communication device suitable for use with various embodiments. [Modes for carrying out the invention]

[0013] Various embodiments are described in detail with reference to the accompanying drawings. Where possible, the same reference numerals are used throughout the drawings to refer to the same or similar parts. References made to specific examples and implementations are illustrative and not intended to limit the scope of the claims.

[0014] The embodiment includes a computing device configured to automatically generate personal responses based on privacy level settings. The computing device may be configured to perform an analysis of the current status of the computing device user (e.g., the current status of the device user) based on diverse privacy-aware multimodal information collected from various information sources and sensors within the device, in order to generate a variety of candidate responses to an incoming communication. By analyzing the collected multimodal information, the computing device may determine the current context of the incoming communication as well as the user's current activity and status. The computing device may use this contextual information in combination with the user's privacy settings to generate appropriate prompts for input to a Generative Large-Scale Language Model (LLM) to generate a number of optional responses.

[0015] To prepare context and other user information for use when generating prompts for the Generative LLM, the computing device may use several information processing submodules received as input sensor or text information to output text in prompts suitable for input to the Generative LLM. The term “submodule” is used herein to refer to subsystems of the computing device and / or dedicated programs running within a processing system configured to analyze a particular type of data or information and generate output in a format suitable for input to the Generative LLM. Some information submodules may receive information from text-based sources such as memory and be configured to format such text commissions into a format suitable for use when generating prompts for the Generative LLM. Other information submodules may receive non-text data such as sensor data (e.g., camera, microphone, accelerometer, etc.) and be configured to interpret the data and generate text-based output suitable for input to the Generative LLM. They interact with each other and / or with other components within the computing system to provide or implement high levels of functionality.

[0016] Some non-limiting examples of several modules that may be used in various embodiments include: a visual understanding submodule configured to recognize a scene or object in a camera image and generate descriptive text; a speech recognition submodule configured to receive sound from a microphone, recognize speech, and generate text (i.e., perform speech-to-text transcription); a sound processing submodule configured to receive sound from a microphone and generate text describing background sounds; an ECG analysis submodule configured to receive electrocardiogram (ECG) data from a sensor (e.g., the user's smartwatch) and generate text describing the user's state as indicated by the ECG data; and a motion analysis submodule configured to process accelerometer and gyroscope information from an inertial measurement unit (IMU) in a computing device and generate text describing the user's movement or activity.

[0017] In various embodiments, the computing device may include a model controller configured to determine the context of an incoming communication and user activity, access the user's privacy settings or preferences, and then control which of various information submodules provides output to the generating LLM. The model controller may be configured to restrict the type of information used by the generating LLM to generate a list of suggested responses, notified by the context of the incoming communication and the user's situation, based on the user's privacy preferences. By enabling or disabling access to information sources provided by probation submodules, the generating LLM may be controlled to generate a list of suggested personalized auto-reply messages that are appropriate for the situation and consistent with the user's privacy preferences. Various embodiments may provide a comprehensive and flexible auto-reply solution that readily adapts to a wide variety of situations and contexts that match the user's privacy preferences / settings.

[0018] The term “computing device” may be used herein to mean any one or all of the following similar devices: personal computers, laptop computers, tablet computers, user equipment (UE), smartphones, personal or mobile multimedia players, personal data assistants (PDAs), palmtop computers, wireless email receivers, multimedia internet-enabled cellular phones, gaming systems (e.g., PlayStation®, Xbox®, Nintendo Switch®, etc.), wearable devices (e.g., smartwatches, head-mounted displays, fitness trackers, etc.), media players (e.g., DVD players, ROKU®, Apple TV®, etc.), digital video recorders (DVRs), automotive displays, portable projectors, 3D holographic displays, and other similar devices including displays and programmable processors that can be configured to provide functions in various embodiments.

[0019] The term “processing system” is used herein to refer to a system comprising one or more processors, including multicore processors, organized and configured to perform a variety of computing functions. A processing system may include at least one memory, interface circuits, and other components integrated into the system. In a processing system, one or more of the processors may be configured to perform one or more operations of various embodiments.

[0020] The term "system on chip" (SoC) as used herein refers to a single integrated circuit (IC) chip that includes multiple processors, at least one memory, and auxiliary resources, and can form an integrated processing system on a single substrate. An SoC may include circuits for digital functions, analog functions, mixed-signal functions, and radio frequency functions. A single SoC processing system may also include any number of general-purpose or dedicated processors (e.g., network processors, digital signal processors, modem processors, video processors, etc.), memory blocks (e.g., ROM, RAM, Flash, etc.), and resources (e.g., timers, voltage regulators, oscillators, etc.). For example, an SoC processing system may include an application processor acting as the main processor of the SoC, a central processing unit (CPU), a microprocessor unit (MPU), an arithmetic logic unit (ALU), etc. An SoC processing system may also include software that controls the integrated resources and processors, as well as software that controls peripheral devices.

[0021] The term "system in a package" (SIP) may be used herein to refer to a single module or package that includes a processing system containing two or more IC chips, substrates, or multiple resources, computing units, cores, or processors on an SoC. For example, a SIP processing system may include a single substrate on which a plurality of IC chips or semiconductor dies are vertically stacked. Similarly, a SIP processing system may include one or more multi-chip modules (MCMs) on which a plurality of ICs or semiconductor dies are packaged within a unified substrate. A SIP processing system may also include multiple independent SOCs coupled to each other via high-speed communication circuitry and packaged in extremely close proximity, such as on a single motherboard, within a single UE, or within a single CPU device. The proximity of the SoCs facilitates high-speed communication and sharing of memory and resources.

[0022] Auto-reply is a function in modern computing devices for generating automated responses or actions in a computer system or software application when a specific event or condition is satisfied (such as when an email or support request is received). Auto-reply solutions can improve user experience by enabling actions to be performed (such as sending a message) without interrupting existing calls. Various embodiments improve computing devices by enhancing the responsiveness and flexibility of auto-reply solutions that align with the context of incoming communications, user activity, and user privacy preference settings. Various embodiments further improve computing devices by learning over time how to generate proposed auto-reply responses that meet user preferences.

[0023] The auto-reply solution may support both message-based and non-message-based responses and actions. Non-message-based responses / actions may include blocking calls from a specific phone number, or allowing the user to press an end button to reject an incoming call. The message-based responses / actions may allow the user to select from an optional list of text messages to send to a caller when the user is engaged in another call. For example, the auto-reply system may select from template response messages such as "I cannot talk right now, please contact me via text" or "I will call you back shortly". Some systems may also allow the user to pre-create and register custom messages such as "I am currently driving" or "I am in a meeting".

[0024] Conventional auto-reply solutions are not sufficiently capable of providing personalized or detailed actions or messages. For example, with conventional solutions, the proposed texts may be pre-created by the user, software, or network provider and stored in memory. Therefore, auto-reply messages are often generic, as they are created to be applicable in a wide variety of possible situations. For example, during an urgent meeting, a template message such as "I will call you back shortly" may not properly convey the importance of the situation. Custom messages offer a certain degree of adaptability, but still do not cover all possible scenarios.

[0025] Similar functionality in some software systems (such as Microsoft Outlook®) can generate suggestions for concise responses as soon as a reply is initiated. For example, suggestions may be generated by an artificial intelligence (AI) system that analyzes the main subject of the incoming email and generates suggested responses based on words in the email. These systems may be constrained by the content of the incoming email and may not be suitable for generating responses tailored to individual users, specific situations, or user privacy preferences.

[0026] Various embodiments include computing devices (e.g., smartphones, tablets, laptop computers, etc.) having a processing system that runs an advanced interaction system (AIS) component that collects information about the user from various device sensors and memory. Information from such an AIS may be processed by a model controller to determine the type of information provided to a generating LLM to automatically generate a menu of personalized response suggestions for incoming communications (e.g., calls, texts, emails, etc.) that respond to the context of the communication, the current user activity, and the user's privacy preferences. In some embodiments, the model controller may be configured to selectively restrict access to the generating LLM's information sources and information submodules so that the generating LLM generates proposed responses based on the context of the incoming communication, the user's current situation, multimodal information collected within the computing device, and the user's privacy preferences or settings for automated reply responses.

[0027] In some embodiments, the model controller is configured to enable or disable the generated LLM's access to the collected multimodal information, such as activating or deactivating various information submodules, based on or in response to the user's privacy preferences or settings (e.g., the user's selected privacy level). In some embodiments, the model controller may determine the user's privacy settings for each distinct category of user information during incoming calls, texts, or emails (optionally via a user menu). In some embodiments, the user's privacy settings may also be pre-configured for each distinct category of user information. In some embodiments, the user's privacy preferences or settings for auto-reply responses may include or identify categories of information that the user allows to be included in auto-reply responses, and the preferences or settings are specified for or based on at least one of the following: the context of the incoming call or message, the caller of the incoming call or message, the user's current location, and / or the user's current activity. In this way, users can pre-specify the types of information that may be included in an automated reply, depending on who is calling / messaging, what the message is about, what is happening to the user at that moment, and other criteria that the user may include in their privacy preferences.

[0028] In some embodiments, the model controller may be configured to detect incoming communication, determine or obtain the user's privacy settings, select information sources and / or information submodule outputs based on the user's privacy settings, generate LLM query information (e.g., prompts for the generating LLM) based on the selected information sources, send the LLM query information to the generating LLM to generate a list or set of proposed customized responses that conform to the user's privacy settings, render the proposed customized responses on an electronic display, and allow the user to select one of the rendered proposed customized responses. The computing device may then send the selected proposed customized response as an auto-reply message and / or use the selected proposed customized response as feedback to a machine learning system or module to learn the user's preferences over time.

[0029] In some embodiments, the AIS components may be configured to determine, characterize, represent, and / or store the user's current situation as an information structure (e.g., User Circumstances) that includes symbols or numerical values ​​representing combinations of multimodal information (e.g., sensor information, location information, calendar insights, etc.).

[0030] In some embodiments, the AIS component may be configured to collect multimodal information from any or all of the various sensors, information submodules, and memory within the computing device. Examples of multimodal information that may be collected in the device include sensor information, identification information, operating mode information, location information, calendar insights, audiovisual data, motion data, health data, connectivity information, data network activity information, system resource usage information, status information, driver statistics, hardware component information, software application information, and transmission information.

[0031] In some embodiments, the AIS component and / or model controller may be configured to determine the user's current status based on collected multimodal information. For example, the AIS component and / or model controller may be configured to detect trigger events (e.g., incoming calls, texts, emails, etc.), collect multimodal information, determine the user's current status based on the collected multimodal information, and provide the collected or determined information to the model controller. The model controller may be configured to use the received information and the user's privacy preferences / settings to determine which information sources should be provided to the generating LLM to generate a list or set of personalized response suggestions. The suggested customized responses received from the LLM may be rendered on an electronic display for user selection. The processing system of the computing device may receive the user selection (e.g., a touch on a touchscreen display of one of the suggested responses) and take action based on the selected response, or similar actions, such as blocking the caller, sending the selected response as an auto-reply message, or formulating and generating an auto-reply message using the selected response.

[0032] In some embodiments, the AIS component, model controller, or another module running within the processing system of the computing device may also use customized responses of user selection as feedback to the machine learning system, allowing the model controller to learn user preferences for generating customized responses and, over time, learn to better satisfy the user with proposed automated response feedback. In some embodiments, the AIS component may control and fine-tune the information input to the generating LLM in accordance with user expectations and privacy controls.

[0033] In some embodiments, the model controller may be configured to select and / or exclude outputs provided by various information submodules and / or multimodal information sources based on the user's privacy settings. For example, the model controller may be configured to use the user's privacy settings to determine which information sources and / or information submodules are appropriate to use when generating customized responses under various contexts and situations in response to the user's privacy preferences, and to provide only the selected multimodal information to the generating LLM in order to generate the proposed automated reply response.

[0034] In some embodiments, the AIS component may be configured to provide the user with options for setting privacy levels for different information categories and / or for generating proposed responses that conform to user-selected privacy levels or settings. Examples of privacy levels include anonymous, public, acquaintance, professional, friend, and family.

[0035] Various embodiments can be implemented in the processing systems of computing devices, which may include several single-processor and multi-processor computer systems, SoC processing systems, or SIP processing systems. Figure 1 shows the architecture of an exemplary SIP processing system 100 that may be used in a mobile computing device implementing various embodiments.

[0036] Referring to Figure 1, the illustrated exemplary SIP processing system 100 includes two SOC processing systems 102 and 104, a clock 106, a voltage regulator 108, and a wireless transceiver 166. The first SOC processing system 102 and the second SOC processing system 104 may communicate via an interconnection bus 150. Various processors 110, 112, 114, 116, 118, 121, and 122 may be interconnected with each other, as well as with one or more memory elements 120, system components and resources 124, and a thermal management unit 132 via an interconnection bus 126, and the interconnection bus 126 may include advanced interconnections such as high-performance network-on-chips (NOCs). Similarly, processor 152 may be interconnected with a power management unit 154, a millimeter-wave transceiver 156, at least one memory 158, and various additional processors 160 via an interconnection bus 164. These interconnect buses 126, 150, and 164 may include arrays of reconfigurable logic gates and / or implement bus architectures (e.g., CoreConnect, AMBA, etc.). Communication may be provided by advanced interconnects such as NOCs.

[0037] In some embodiments, the first SOC processing system 102 may operate as the central processing unit (CPU) of a mobile computing device, executing instructions of a software application program by performing arithmetic, logical, control, and input / output (I / O) operations specified by the instructions. In some embodiments, the second SOC processing system 104 may operate as a dedicated processing unit. For example, the second SOC processing system 104 may operate as a dedicated 5G processing unit, responsible for managing high-capacity, high-speed (e.g., 5 Gbps) and / or extremely high-frequency and short-wavelength (e.g., 28 GHz mmWave spectrum) communications.

[0038] The first SOC processing system 102 may include a digital signal processor (DSP) 110, a modem processor 112, a graphics processor 114, an application processor 116, one or more coprocessors 118 (e.g., vector coprocessors) connected to one or more of the processors, at least one memory 120, a deep processing unit (DPU) 121, an artificial intelligence processor 122, system components and resources 124, an interconnection bus 126, one or more temperature sensors 130, a thermal management unit 132, and a thermal power envelope (TPE) component 134. The second SOC processing system 104 may include a 5G modem processor 152, a power management unit 154, an interconnection bus 164, multiple millimeter-wave transceivers 156, at least one memory 158, and various additional processors 160 such as an application processor and a packet processor.

[0039] Each processor 110, 112, 114, 116, 118, 121, 122, 121, 122, 152, 160 in processing systems 100, 102, 104 may include one or more cores, and each processor / core may operate independently of other processors / cores. For example, the first SOC processing system 102 may include a processor running a first type of operating system (e.g., FreeBSD, LINUX, OS X®, etc.) and a processor running a second type of operating system (e.g., MICROSOFT WINDOWS® 11). Furthermore, any or all of the processors 110, 112, 114, 116, 118, 121, 122, 121, 122, 152, 160 may be included as part of a processor cluster architecture (e.g., a synchronous processor cluster architecture, an asynchronous or heterogeneous processor cluster architecture, etc.).

[0040] Any or all of processors 110, 112, 114, 116, 118, 121, 122, 121, 122, 152, and 160 may operate as the CPU of a mobile computing device. In addition, any or all of processors 110, 112, 114, 116, 118, 121, 122, 121, 122, 152, and 160 may be included as one or more nodes in one or more CPU clusters. A CPU cluster can be a group of interconnected nodes (e.g., processing cores, processors, SOCs, SIPs, computing devices, etc.) configured to work together to perform computing tasks. Each node runs its own operating system and may include its own CPU, memory, and storage. Tasks assigned to a CPU cluster may be divided into smaller tasks distributed across individual nodes for processing. Nodes may cooperate to complete tasks, with each node processing a portion of the computation. The results of the computations from each node may be combined to produce a final result. CPU clusters are particularly useful for tasks that can be executed in parallel and simultaneously. This allows CPU clusters to complete tasks much faster than a single high-performance computer. Furthermore, because CPU clusters consist of multiple nodes, they are often more reliable and less prone to failure than a single high-performance component.

[0041] The first SOC processing system 102 and the second SOC processing system 104 may include various system components, resources, and custom circuits for managing sensor data, analog-to-digital conversion, wireless data transmission, and other specialized operations such as decoding data packets and processing encoded audio and video signals for rendering in a web browser. For example, the system components and resources 124 of the first SOC processing system 102 may include power amplifiers, voltage regulators, oscillators, phase-locked loops, peripheral bridges, data controllers, memory controllers, system controllers, access ports, timers, and other similar components used to support processors and software clients running on mobile computing devices. The system components and resources 124 may also include circuits for interface with peripheral devices such as cameras, electronic displays, wireless communication devices, and external memory chips.

[0042] The first and / or second SOC processing systems 102, 104 may further include input / output modules (not shown) for communicating with resources outside the SOC, such as a clock 106, a voltage regulator 108, and a wireless transceiver 166 (e.g., a cellular wireless transceiver, a Bluetooth transceiver, etc.). The external resources of the SOC (e.g., the clock 106, the voltage regulator 108, the wireless transceiver 166) may be shared by two or more of the internal SOC processors / cores.

[0043] In addition to the exemplary SIP processing system 100 described above, various embodiments can be implemented in a wide variety of computing systems, which may include a single processor, multiple processors, a multi-core processor, or any combination thereof.

[0044] Figures 2A and 2B show the logical configuration of exemplary components in a computing system suitable for implementing various embodiments (e.g., SIP processing system 100, SOC processing system 102, etc.). Referring to Figures 1 to 2B, the computing system may include a model controller 202, an LLM 204 component (e.g., a generated LLM), a visual understanding 212 submodule, a speech recognition 214 submodule, an electrocardiogram (ECG) analysis 216 submodule, a motion analysis 218 submodule, a display module 220, and a user input module 222. The functions of the model controller 202 and the various information submodules 212 to 218 can be performed in processing systems such as processing systems 100, 102, and 104, as described with reference to Figure 1.

[0045] The model controller 202 may act as a gatekeeper, restricting the output of information submodules 212-218 and other data sources provided to the LLM 204 for generating multiple proposed auto-reply responses. The model controller 202 may be configured to analyze multimodal information to determine the context of the incoming communication and the user's situation, and to control the generating LLM's access to information submodules 212-218 and other information sources (for example, by activating or deactivating selected submodules) in response to the user's privacy preferences or settings regarding auto-reply responses.

[0046] The model controller 202 may be configured to receive a user ID and privacy preferences / settings indicating categories of data that may be used by the user as input. The model controller 202 may also collect or receive multimodal data which may include text-based and non-text information. In some embodiments, the model controller 202 may determine submodules 212-218 that should be activated to filter the collected information, and may use gating to activate or deactivate the determined submodules so that the output of the model controller 202 flows through the activated submodules 212-218 to the LLM 204. The LLM 204 may generate response suggestions based on the received outputs of submodules 212-218.

[0047] The model controller 202 can also collect or receive text-based and non-text information stored in the computing device's memory and / or received from various sensors on or coupled to the computing device. In some embodiments, the model controller 202 can receive information via submodules 212-218. Examples of text-based information include user location (from GPS), contact data (name, relationship to the user), calendar data (upcoming events and meetings), connectivity data (Bluetooth, Wi-Fi status), and the user's current activity (from operating system information). Examples of non-text information include environmental data (e.g., sound data from a microphone, visual data from a camera system) and user health data (e.g., from a smartwatch). In general, text-based information can be transmitted to the LLM204 without further analysis. Non-text information (e.g., received from various sensors) may require analysis and conversion to text before being input to the LLM204. Various submodules 212-218 may perform such additional multimodal analysis operations to convert non-text information into a format suitable for use as input to LLM204.

[0048] In some embodiments, various information submodules 212-218 may include two or more models, each having a different size or computing power. Models of different sizes may be capable of analyzing different amounts of information. In such embodiments, the model controller 202 may be configured to determine an appropriate model (e.g., an appropriate model size) within each activated submodule, taking into account the determined situation, the nature of the incoming message or phone call, the origin of the incoming message or phone call, etc. For example, activating a small model in the visual understanding 212 submodule may result in the system analyzing only the types of objects present in the image, which may be applicable to responding to several incoming messages. By activating a larger model, the system may use image captioning techniques to generate extensive text describing the relationships between objects in the image, which may be appropriate for providing a detailed response to an incoming message. For example, in response to determining that the information that can be generated by the Visual Understanding 212 submodule is important for prompting LLM204 in the current context, the Model Controller 202 may activate the larger visual processing model within the Visual Understanding submodule 212 to perform a more robust visual analysis of the data and send additional information to LLM204.

[0049] In some embodiments, the model controller 202 may be configured to operate in rule-based mode (without learning) or as a small LLM. In rule-based mode, the model controller 202 may open specific model gates based on given circumstances. Different individuals may desire different responses in the same situation, and the desired level of privacy for the response may vary for the same person depending on the specific circumstances. For example, a user's privacy preferences may specify categories of information that the user allows to be included in an auto-reply response, depending on variables such as the context of the incoming call or message, the caller of the incoming call or message, and the user's location and / or current activity. Thus, the model controller 202 may use machine learning techniques to learn from the generated text from the LLM 204 and the feedback related to the user's selected response, thereby improving the LLM's selective access to different information sources and information processing submodules 212-218 based on the incoming communication, the user's situation, and personal preferences.

[0050] The Visual Understanding submodule 212 may be configured to analyze visual information and generate text describing several aspects of an imaged scene. For example, the Visual Understanding 212 submodel may perform image recognition, which may include classifying objects, people, animals, etc., in an image. The Visual Understanding 212 submodel may perform semantic segmentation, which may include analyzing detailed pixels of an image to identify the locations and boundaries of different object classes. The Visual Understanding 212 submodel may also use image captioning, which may consider objects, backgrounds, and their relationships in an image to generate more meaningful text descriptions suitable for input to LLM204.

[0051] The speech recognition submodule 214 may be configured to decode and interpret auditory signals, convert spoken words into text, and convert such information into text format to facilitate voice-controlled interaction. For example, the speech recognition submodule 214 may be configured to recognize and analyze audio information from the device's microphone, analyze the input audio signal, and convert the audio information into a text description based on the analysis. The speech recognition submodule 214 may transcribe the audio or determine the current situation based on the input sound (e.g., identify the speaker, analyze the speaker's emotions or mood). Similarly, a speech recognition submodule (not shown separately) may be configured to analyze the sound received by the microphone to determine the nature of the background sound and the generated text description of the ambient sound picked up by the microphone.

[0052] The ECG analysis submodule 216 may be configured to interpret and evaluate electrical signals corresponding to the user's cardiac activity, provide insights into the user's health or emotional state, and convert sensor data into information in a form (e.g., text) that can be received and processed by the LLM 204. Such ECG analysis may be performed via a smartwatch or other dedicated electrical device that enables the analysis of electrical signals. The ECG analysis submodule 216 may also be configured to enable an understanding of the cardiac electrical signals, enabling the calculation of heart rate and the detection of anomalies associated with various cardiac conditions.

[0053] The motion analysis submodule 218 may be configured to evaluate motion sensor data (e.g., physical motion captured via an accelerometer, gyroscope, etc.) and convert the sensor data into information in a format (e.g., text) that can be received and processed by the LLM 204. For example, the motion analysis submodule 218 may analyze a user's movements based on motion information received via a smartwatch or other electronic device. Based on the motion data, the motion analysis submodule 218 may determine whether the user is moving or stationary. The motion analysis 218 submodel may analyze a specific action or activity of the user based on the user's movement patterns.

[0054] The LLM204 component may be configured to receive prompt input from the model controller 202 and / or selected information submodules 212-218, and to use that prompt to generate a list of suggested personalized responses that respond to matched multimodal information, the user's current context or activity, and the user's privacy preferences or settings for automated reply responses. The LLM204 component may receive text input generated by the model controller 202 and / or selected information submodules 212-218 based on the results of collected, analyzed, transformed, and filtered text-based and non-text information from the model controller 202. The LLM204 may analyze the received information to generate several nuanced personalized responses that effectively take into account the user's current situation and privacy settings.

[0055] The display module 220 may be configured to render a proposed personalized response and accept user input to select a preferred automated response option. For example, the display module 220 may be a touchscreen display, such as on a smartphone.

[0056] The user input module 222 may be configured to receive and process user selections or inputs, and these selections or inputs may be fed back into the system for continuous improvement and learning.

[0057] As an example, the processing system may receive as input the privacy preferences / settings of a specific user, which may be stored in memory. The processing system may use the user's privacy preferences / settings input to make decisions regarding the selective activation or deactivation of LLM access to various information submodules 212-218 (e.g., by selectively activating or deactivating them) based on the user's privacy settings and the context of incoming communications and current user activity. For example, the user input may include the user's privacy settings, which set permissions for location, calendar, camera, and microphone to "ON" and permissions for Bluetooth, health, ECG, and motion to "OFF". In response, the model controller 202 may include the outputs from submodules 212-218 in the data sent to the LLM 204, providing text related to the user's location, upcoming calendar events, visual information from the camera, and auditory data from the microphone, while excluding information from the Bluetooth connection, health monitor, ECG sensor, and motion detector.

[0058] As another example, the model controller 202 may be configured to determine data relevant to an incoming communication. The model controller 202 may process multimodal data without feedback, additional training, or updates. The model controller 202 may determine the data used for multimodal analysis based on the user's privacy preferences or settings for automated reply responses. Text-based information (e.g., location information indicating the user is in "Building A203," or calendar information indicating an "academic seminar from 10am to 11am") may be sent to the LLM204 components without extensive processing or analysis. The model controller 202 and / or submodules 212-218 may perform multimodal analysis on non-text information (e.g., visual and auditory data from microphones and cameras) to convert the non-text information into a text format suitable for input to the LLM204. The LLM204 may analyze the visual and auditory data in conjunction with text-based inputs such as location and calendar events to generate a contextually relevant response that conforms to the user's privacy settings.

[0059] In some embodiments, the model controller 202 may collect and filter multimodal information from different submodules based on the user's privacy preferences / settings to create a robust context-aware understanding of the user's context. By selecting different filtering mechanisms based on the user's privacy settings, the model controller 202 may control the information used to generate the proposed response. For example, the visual understanding 212 submodule may determine that there is a lack of visual input or a black screen, which is likely due to the computing device being in the user's pocket. In response, the visual understanding 212 submodule may generate the text “No visual information detected” which is sent to the LLM204 component. Meanwhile, the speech recognition submodule 214 may detect and analyze the speaker's voice to understand that the speaker is currently giving a presentation. Based on this auditory data, the LLM204 may generate a proposed auto-responder stating “Multiple individuals, including the phone owner, are involved in the conversation, and the topic of the conversation is related to deep learning.”

[0060] In some embodiments, the system combines text-based and non-text information to generate a comprehensive understanding, thereby enabling the LLM204 components to generate a variety of response options that relate to the incoming communication, respond to the user's current situation, and respect the user's privacy preferences. The generated automated reply options can range from generic responses to highly detailed responses that reflect all available relevant information. For example, text-based information such as location details ("A203 building") and calendar events ("Academic seminar from 10am to 11am") may be combined with non-text information such as visual understanding ("No visual information detected.") and speech recognition ("Multiple individuals, including the phone owner, are involved in the conversation, and the topic of the conversation relates to deep learning."). In response to such input, the LLM204 components can generate a range of nuanced response options, from generic responses that do not utilize any specific information to highly detailed responses that incorporate all available data. Examples of such options may include: "I can't answer the phone right now," "I'm currently in building A203 and will contact you later," "I'm currently in building A203 and attending an academic seminar from 10am to 11am. I'll contact you later," "I'm currently in building A203 and attending an academic seminar from 10am to 11am. My phone is in my pocket and I can't check it right now. I'll contact you later," and "I'm currently in building A203 and attending an academic seminar from 10am to 11am. My phone is in my pocket and I'm currently talking to several people about deep learning. I'll contact you later."

[0061] In some embodiments, the model controller may be retrained based on a specific response selected by the user to generate better response options for subsequent incoming communications. This retraining may be performed repeatedly or continuously to fine-tune the operation, better align with user preferences, and / or improve the performance of the auto-reply function. For example, one user may prefer to share a large amount of information in certain situations, while another user may prefer to share more limited personal information. Therefore, the model controller may, in response to determining that a device user is participating in a meeting based on calendar and location data, activate both the visual understanding submodule and the speech recognition submodule to analyze the information collected by the camera and microphone. The model controller may send the collected information to the LLM, present options to the user, evaluate the selected option, and decide whether to activate more submodules 212-218, use a larger model to provide more detailed information, or deactivate modules and use a smaller model to minimize resource usage.

[0062] Figure 3 shows a method 300 for performing an automated reply in a computing device according to several embodiments. Referring to Figures 1 to 3, the operation of method 300 may be performed in a computing device by a processing system (e.g., 100, 102, 104) which may include one or more processors (e.g., processors 110, 112, 114, 116, 118, 121, 122, 121, 122, 152, 160, etc.), components, or subsystems (e.g., model controller 202, submodules 212-218, LLM204, etc.). Furthermore, one or more processors within the processing system may be configured to perform various operations of the method by software or firmware. The elements that perform the operation of the method, encompassing any of the processors (one or more), hardware elements, and software elements that may be involved in the execution of method 300, are generally referred to as the “processing system”. Means for performing the functions of Method 300 may include a processing system (e.g., 100, 102, 104) including one or more processors (e.g., processors 110, 112, 114, 116, 118, 121, 122, 121, 122, 152, 160, etc.), components, or subsystems (e.g., model controller 202, submodules 212-218, LLM204, etc.) as described herein.

[0063] In block 302, the processing system may detect trigger events such as incoming calls, incoming texts, and incoming emails. For example, the processing system may detect an incoming call by monitoring signaling information of a telecommunications network, or detect an incoming text or email by checking for new messages on a specific application or subsystem. The processing system may also be configured to recognize other trigger events such as calendar alerts or reminders by interacting with scheduling and time management applications.

[0064] In block 304, the processing system may collect multimodal information related to the user from multiple information sources on or accessible by the computing device. For example, the processing system may collect sensor information from embedded sensors that measure attributes such as position, motion, temperature, and pressure. The processing system may collect location information from a global navigation satellite system (GNSS) module (e.g., a Global Positioning System (GPS) module) that accurately indicates the geographical location of the device. The processing system may collect calendar information from scheduling applications, audiovisual data from microphones and cameras, motion data from accelerometers, health data from health monitoring components such as heart rate sensors, connectivity information, data network activity information, system resource usage information, etc., through corresponding hardware and software components that monitor and control the following subsystems within the computing device. The processing system may also collect user information stored in memory, such as the user's age, gender, marital status, education level, employment status, and employer. By aggregating various multimodal pieces of information, processing systems can build an understanding of certain nuances in the user's current situation, enabling more personal, context-aware interactions.

[0065] In block 306, the processing system may determine the current user status based on the collected multimodal information. For example, the processing system may combine sensor readings, location data, calendar insights, motion information, and connectivity details to determine the user's context. As a more detailed example, the processing system may use location information combined with calendar data to determine that the user is currently in a scheduled meeting. The processing system may use motion data and sensor information to determine whether the user is currently driving, running, or walking. The processing system may use connectivity information to determine whether the device is connected to a home network or an office network, and therefore whether the user is currently located at home or in the office.

[0066] In block 308, the processing system may determine the user's privacy preferences / settings. In some embodiments, the processing system may determine the user privacy level based on a user identifier and / or user input that identifies categories of data that may be used by the user. In some embodiments, the user's privacy preferences or settings for auto-reply responses may be stored in memory, and the processing system may determine the user's privacy preferences / settings by accessing that memory. In some embodiments, the user's privacy preferences for auto-reply responses include information that identifies categories of information that the user may allow to be included in auto-reply responses based on, or in response to, the context of the incoming call or message, the caller of the incoming call or message, the user's current location, or the user's current activity. In some embodiments, the processing system may determine the user's privacy level based on any or all of the user's preferences, pre-configured settings, and contextual analysis. For example, a user may define privacy levels across different information categories and select preferences for various conditions or features such as location sharing, access to personal data, or communication permissions. Users may set such preferences for different individuals, contexts, or relationships, such as spouse, family, occupation, or public privacy preferences. Furthermore, the processing system may allow users to preconfigure privacy settings for distinct categories of information, providing fine-grained control over what is shared with whom. The processing system may also dynamically adjust privacy settings based on the user's current situation and / or based on predefined rules or user behavior patterns.

[0067] In block 310, the processing system performing the model controller function may enable or disable access to information sources by the generated LLM, for example, by selectively activating or deactivating information submodules based on the current user situation and the user's privacy preferences or settings regarding the auto-response response. For example, if the user's privacy settings allow location, calendar, camera, and microphone data to be included in the auto-response response but restrict the disclosure of Bluetooth, health, ECG, and motion data in the auto-response response, the processing system may selectively activate information submodules corresponding to or related to the permitted information categories and deactivate information submodules related to the restricted information categories. In some embodiments, the complexity and / or size of the analysis of the language model within each information submodule may be selected based on the current user situation and the user's privacy settings. In some embodiments, activating or deactivating information submodules may involve using a selective gate mechanism within the computing device.

[0068] In block 312, the processing system may generate prompts suitable for input to the generated LLM by applying relevant subsets of the collected multimodal information to an active information submodule. For example, the processing system may apply relevant subsets of the collected multimodal information, including both text-based and non-text-based data, to an active information submodule. As part of these operations, the processing system may filter and / or guide the collected information through the activated information submodule. The processing system may process non-text-based information, such as audio or video data, through dedicated information submodules designed for auditory or visual analysis. For example, an audio understanding submodule may interpret spoken words or background sounds, and a visual understanding submodule may analyze objects and relationships in a video or image to generate text suitable for any LLM prompt.

[0069] In block 314, the processing system may input the generated prompt to the generating LLM. In various embodiments, the processing system may send an LLM prompt to a generating LLM hosted locally within the device, or to add an LLM service host remotely on a server. For example, after generating an LLM prompt by applying both text-based and non-text-based information to an active submodule within the computing device, the processing system may package the resulting output as an LLM prompt to be sent to the generating LLM. This transmission can be facilitated over a secure communication channel that ensures data integrity and confidentiality. In response to receiving the prompt, the generating LLM may generate a list of personalized response suggestions based on the input.

[0070] In block 316, the processing system may receive a list of personalized response suggestions from the generated LLM. The list of personalized response suggestions may include a range of nuanced response options, from general responses that do not utilize any specific information to very detailed responses that incorporate all available data. Examples of such options may include: "I am currently unable to answer the phone," "I am currently in building A203 and will contact you later," "I am currently in building A203 and attending an academic seminar from 10am to 11am. I will contact you later," "I am currently in building A203 and attending an academic seminar from 10am to 11am. My phone is in my pocket and I cannot check it now. I will contact you later," and "I am currently in building A203 and attending an academic seminar from 10am to 11am. My phone is in my pocket and I am currently talking to several people about deep learning. I will contact you later."

[0071] In block 318, the processing system may render the proposed personal responses on an electronic display for user selection. The rendering operation may include converting the received suggestions into a format suitable for visual presentation and organizing them into a user-friendly interface. The processing system may display the responses in a list, grid, or other visually appealing layout, accompanied by interactive elements that allow the user to easily select the desired response.

[0072] In block 320, the processing system may receive input to select one of the rendered proposed personal responses. For example, the user may select an option via a touchscreen interface by tapping the desired response, or via a mouse click when using a desktop or laptop. User input may also be a voice command or gesture.

[0073] In block 322, the processing system may perform an automated reply action based on the selected response (e.g., blocking the caller, sending the selected response as an automated reply message, formulating and generating an automated reply message using the selected response). For example, if the selected response is to block the caller, the processing system may update the blocking list or use specific call control functions to prevent further communication from that number. If the selected option is to send an automated reply message, the processing system may automatically create a text or multimedia message based on the selected response and send it to the original caller / sender via the appropriate channel (e.g., SMS, email). The processing system may also use the selected response as a basis to generate a more detailed automated reply message that integrates additional information or tailors the content to the selected option. The processing system may also perform an automated reply action using templates, scripts, or further processing by generating LLM and / or AI components. In some embodiments, the processing system may perform an automated reply action based on predefined rules, user preferences, and / or real-time context.

[0074] In some embodiments, in block 322, the processing system may, taking into account the context and caller of the incoming message or call, use the user's privacy preferences for the auto-reply response to generate a privacy-aware multimodal generated auto-reply message based on the received user input, and send the generated privacy-aware multimodal generated auto-reply message to the computing device that initiated the incoming call or message.

[0075] In the optional block 324, the processing system may send the selected response as feedback to a machine learning module to support the training of the model controller and allow it to learn the user's preferences over time. This feedback or retraining loop may be part of an adaptive system that continuously improves and personalizes the system's behavior and recommendations. For example, the processing system may analyze specific responses that the user selects over time and identify patterns, preferences, and / or tendencies that reflect the user's behavior and decision-making. The processing system may use any or all such information to tune or retrain the models within the system. For example, if the user consistently selects a particular type of response for a given scenario, the machine learning system may learn to prioritize or recommend similar responses in future interactions.

[0076] Various embodiments (including, but not limited to, the embodiments described above with reference to Figures 1 to 3) can be implemented in a wide variety of wireless devices and computing systems, including a laptop computer 400, an example of which is shown in Figure 4. Referring to Figures 1 to 4, the laptop computer may include a processor 402 coupled to at least one memory, such as volatile memory 404, and a large-capacity non-volatile memory, such as a Flash memory disk drive 406. The laptop computer 400 may include a touchpad touch surface 408 that functions as the computer's pointing device and is therefore capable of receiving drag, scroll, and flick gestures. In addition, the laptop computer 400 may have one or more antennas 410 for transmitting and receiving electromagnetic radiation, which may be connected to a wireless data link and / or cellular telephone transceiver 412 coupled to the processor 402. The computer 400 may also include a BT transceiver 414, a compact disc (CD) drive 416, a keyboard 418, and a display 420, all coupled to the processor 402. Other configurations of the processing system may include, as is well known, a computer mouse or trackball coupled to the processor (for example, via a Universal Serial Bus (USB) input), which can also be used in conjunction with various embodiments.

[0077] Figure 5 is a block diagram of the components of a computing device 500 suitable for use with various embodiments. Referring to Figures 1 to 5, various embodiments can be implemented on various computing devices 500, one example of which is shown in Figure 5 in the form of a smartphone. The computing device 500 may include a first SoC processing system 102 coupled to a second SoC processing system 104. The first SoC processing system 102 and the second SoC processing system 104 may be coupled to at least one internal memory 516, a display 512, and a speaker 514. The first SoC processing system 102 and the second SoC processing system 104 may also be coupled to at least one subscriber identity module (SIM) 540 and / or SIM interface capable of storing information supporting a first 5G NR subscription and a second 5G NR subscription supporting services on a 5G non-standalone (NSA) network.

[0078] The computing device 500 may include an antenna 504 for transmitting and receiving electromagnetic radiation, which may be connected to a wireless transceiver 166 coupled to one or more processors in the first SOC processing system 102 and / or the second SOC processing system 104. The computing device 500 may also include a menu selection button or rocker switch 520 for receiving user input.

[0079] The computing device 500 also includes an audio encoding / decoding (CODEC) circuit 510 that digitizes sound received from a microphone into data packets suitable for wireless transmission, and decodes the received sound data packets to generate an analog signal provided to a speaker for sound generation. In addition, one or more of the processors, wireless transceivers 166, and codecs 510 in the first processing system 102 and the second processing system 104 may include digital signal processor (DSP) circuits (not shown individually).

[0080] The processing system and the included processor or processing unit may be any programmable microprocessor, microcomputer, or one or more multiple processor chips that can be configured by software instructions (applications) to perform a variety of functions, including the functions of the various embodiments described. Some computing devices may have multiple processors, such as one processor in a first circuit dedicated to wireless communication functions and one processor in a second circuit dedicated to running other applications. Software applications may be stored in memory before being accessed and loaded into the processor of the processing system. The processor may have sufficient internal memory to store application software instructions.

[0081] Implementation examples are described in the following paragraphs. Some of the following implementation examples are described in terms of exemplary methods, but further exemplary implementations may include exemplary methods discussed in the following paragraphs, which are implemented by a computing device having at least one memory coupled to at least one processor configured to perform the operation of the methods of the following implementation examples (e.g., by processor-executable instructions), and exemplary methods discussed in the following paragraphs, which are implemented by a computing device having means for performing the functionality of the methods of the following implementation examples, and which can be implemented as a non-temporary processor-readable storage medium storing processor-executable instructions configured to cause the processor of the computing device to perform the operation of the methods of the following implementation examples.

[0082] Example 1. A method for executing an automated reply response by a processing system of a computing device, comprising: collecting multimodal information about the user of the computing device; determining the current user status based on the collected multimodal information; determining the user's privacy preferences for the automated reply response; generating a prompt based on the selected multimodal information and the user's privacy preferences for the automated reply response, inputting the prompt into a generating large-scale language model (LLM); receiving a list of personal response suggestions from the generating LLM; receiving a user input selection of one of the received personal response suggestions in response to rendering the received personal response suggestions on the computing device's electronic display; and executing an automated reply action based on the received user input. Methods that include...

[0083] Example 2. The method according to Example 1, further comprising activating or deactivating one or more information submodules configured to receive data input and output text suitable for use when prompting the generating LLM, based on the determined current user status and the user's privacy preferences for the determined automated response.

[0084] Example 3. The method according to Example 2, further comprising processing non-text-based information by at least one of the active information submodules in order to generate text suitable for input to the generating LLM.

[0085] Example 4. The method according to Example 3, wherein generating a prompt includes generating a prompt by combining text-based information and non-text-based information.

[0086] Example 5. The method according to Example 3, wherein the non-text-based information includes a description based on audio or video sensor data.

[0087] Example 6. The method according to Example 1, further comprising selecting a model size to be used in an information submodule to process non-text-based information based on one or more of the determined current user status, the context of the incoming communication, or the user's privacy preferences for automated reply responses.

[0088] Example 7. The method according to Example 1, wherein the automatic reply action performed based on received user input includes generating a privacy-aware multimodal generated automatic reply message based on received user input, and sending the generated privacy-aware multimodal generated automatic reply message to the computing device that initiated the incoming call or message.

[0089] Example 8. The method according to Example 1, wherein determining the user's privacy preferences for an automated reply response includes determining the user's privacy preferences for an automated reply response based on information that identifies categories of information that the user may allow to be included in the automated reply response, based on at least one of the following: the context of the incoming call or message, the caller of the incoming call or message, the user's current location, or the user's current activity.

[0090] Example 9. The method according to Example 1, further comprising providing a machine learning module with a user input selection of one of the received personal response suggestions to enable improved prompt generation based on selected multimodal information and the user's privacy preferences for the automated response.

[0091] As used in this application, the terms “component,” “module,” and “system” include, but are not limited to, computer-related entities such as hardware, firmware, hardware-software combinations, software, or running software that are configured to perform a particular operation or function. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. For example, both an application running on a computing device and the computing device itself may be referred to as a component. One or more components may reside within a process and / or an execution thread, and components may be localized on one processor or core and / or distributed across two or more processors or cores. Furthermore, these components may be executed from various non-temporary computer-readable media in which various instructions and / or data structures are stored. Components may communicate by local and / or remote processes, function calls or procedure calls, electronic signals, data packets, memory reads / writes, and other known network, computer, processor, and / or process-related communication methods.

[0092] Several different types of memory and memory technologies are available or planned for the future, and any or all of them may be included in and used in systems and computing devices implementing various embodiments. Such memory technologies / types may include magnetoresistive RAM (M-RAM), resistive random access memory (ReRAM or RRAM), phase-change random-access memory (PC-RAM, PRAM, or PCM), ferroelectric RAM (F-RAM), spin-transfer torque magnetoresistive random-access memory (STT-MRAM), and non-volatile random-access memory (NVRAM), such as three-dimensional cross point (3D-XPOINT) memory. Such memory technologies / types may also include non-volatile or read-only memory (ROM) technologies, such as programmable read-only memory (PROM), field programmable read-only memory (FPROM), and one-time programmable non-volatile memory (OTP NVM).Such memory technologies / types may further include volatile random-access memory (RAM) technologies, such as dynamic random-access memory (DRAM), double data rate (DDR) synchronous dynamic random-access memory (DDR SDRAM), static random-access memory (SRAM), and pseudostatic random-access memory (PSRAM). Systems and computing devices implementing various embodiments may also include or use electronic (solid-state) non-volatile computer storage media, such as FLASH memory. Each of the memory technologies described above includes elements suitable for storing instructions, programs, control signals, and / or data in, for example, an advanced driver assistance system (ADAS), a system-on-a-chip (SOC), or other electronic components of a vehicle, or for use therein. Any references to terms and / or technical details relating to individual types of memory, interfaces, standards, or memory technologies are for illustrative purposes only and, unless explicitly stated in the language of the claims, do not limit the claims to any particular memory system or technology.

[0093] The various embodiments illustrated and described are provided merely as examples to illustrate various features of the claims. However, features illustrated and described with respect to any given embodiment are not necessarily limited to the embodiment associated therewith, and may be used in conjunction with or combined with other embodiments illustrated and described. Furthermore, the claims are not intended to be limited by any one exemplary embodiment. For example, one or more of the operations of the Method may be replaced by or combined with one or more operations of the Method.

[0094] The above-described method and process flow diagram are provided merely as illustrative examples and are not intended to require or imply that the operations of the various embodiments must be performed in the order presented. As will be understood by those skilled in the art, the order of operations in the above-described embodiments can be performed in any order. Words such as "t hereafter," "then," and "next" are not intended to limit the order of operations, and these words are used simply to guide the reader through the description of the method. Furthermore, any reference to a claim element in the singular form, for example using the articles "a," "an," or "the," should not be interpreted as limiting that element to the singular form.

[0095] The various exemplary logic blocks, modules, circuits, and algorithmic operations described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware-software compatibility, various exemplary components, blocks, modules, circuits, and operations have been described above in general terms of their functionality. Whether such functions are implemented as hardware or executed as software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as resulting in a departure from the claims.

[0096] The hardware used to implement the various exemplary logics, logic blocks, modules, and circuits described in relation to the embodiments disclosed herein may be implemented or run using a processing system which may include a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate logic or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some operations or methods may be performed by circuits specialized for a given function.

[0097] In one or more embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a non-temporary computer-readable medium or a non-temporary processor-readable medium. The operation of the methods or algorithms disclosed herein may be embodied in a processor-executable software module that may reside on a non-temporary computer-readable storage medium or a non-temporary processor-readable storage medium. The non-temporary computer-readable storage medium or a non-temporary processor-readable storage medium may be any storage medium accessible by a computer or processor. Such a non-temporary computer-readable medium or a non-temporary processor-readable medium may include, but are not limited to, RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the program code in the form of instructions or data structures and may be accessed by a computer. Disks and discs, as used herein, include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, while a disc reproduces data optically using a laser. The above combinations also fall within the scope of non-temporary computer-readable media and non-temporary processor-readable media. In addition, the operation of a method or algorithm may exist as one of the codes and / or instructions on a non-temporary processor-readable medium and / or non-temporary computer-readable medium, or any combination or set thereof, which may be incorporated into a computer program product.

[0098] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to construct or use the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the claims. Therefore, this disclosure is not intended to be limited to the embodiments shown herein, but should be given the broadest scope that is consistent with the following claims and the principles and novel features disclosed herein.

Claims

1. A method for performing an automated reply response by a computing device's processing system, Collect multimodal information about the user of the computing device, Based on the collected multimodal information, the current user status is determined, Determining the user's privacy preferences regarding automated reply responses, The process involves generating a prompt based on the selected multimodal information and the user's privacy preferences regarding the automated response, and inputting the prompt into a Large-Scale Language Model (LLM). Receiving a list of personal response proposals from the aforementioned generated LLM, In response to rendering the received personal response proposals on the electronic display of the computing device, the system receives a user input selection from among the received personal response proposals. The system performs an automated reply action based on the user input received, Methods that include...

2. The method according to claim 1, further comprising activating or deactivating one or more information submodules configured to receive data input and output text suitable for use when prompting the generating LLM, based on the determined current user status and the user's privacy preferences for the determined automated response.

3. The method according to claim 2, further comprising processing non-text-based information by at least one of the active information submodules in order to generate text suitable for input to the generating LLM.

4. The method according to claim 3, wherein generating the prompt includes generating the prompt by combining text-based information and non-text-based information.

5. The method according to claim 3, wherein the non-text-based information includes a description based on audio or video sensor data.

6. The method according to claim 1, further comprising selecting a model size to be used in an information submodule to process non-text-based information based on one or more of the determined current user status, the context of the incoming communication, or the user's privacy preferences for the automated reply response.

7. The automatic reply action is performed based on the received user input. Based on the received user input, a privacy-aware, multimodal automated reply message is generated. The generated privacy-aware multimodal automated reply message is sent to the computing device that initiated the incoming call or message. The method according to claim 1, including the method described in claim 1.

8. The method according to claim 1, wherein determining the user's privacy preferences for the automated reply response includes determining the user's privacy preferences for the automated reply response based on information identifying a category of information that the user may allow to be included in the automated reply response, based on at least one of the context of the incoming call or message, the caller of the incoming call or message, the user's current location, or the user's current activity.

9. The method according to claim 1, further comprising providing the user input selection of one of the received personal response suggestions to a machine learning module in order to enable the generation of the prompt to be improved based on selected multimodal information and the user's privacy preferences for the automated response.

10. A computing device, At least one memory, The display and A processing system comprising at least one memory and the display, and comprising one or more processors, wherein the one or more processors Collect multimodal information about the user of the computing device, Based on the collected multimodal information, the current user status is determined. Determine the user's privacy preferences regarding automated reply responses. A prompt is generated based on the selected multimodal information and the user's privacy preferences for the automated response, and the prompt is input into the Generating Large-Scale Language Model (LLM). A list of personal response proposals is received from the aforementioned generated LLM, In response to rendering the received personal response proposals on the display of the computing device, one user input selection from the received personal response proposals is received. Based on the user input received, an automated reply action is executed. It is structured in such a way. Computing device.

11. The computing device according to claim 10, wherein one or more of the processors of the processing system are further configured to activate or deactivate one or more information submodules configured to receive data input and output text suitable for use when prompting the generating LLM, based on the determined current user status and the user's privacy preferences for the determined automated response.

12. The computing device according to claim 11, wherein one or more of the processors of the processing system are further configured to process non-text-based information by at least one of the active information submodules in order to generate text suitable for input to the generating LLM.

13. The computing device according to claim 12, wherein one or more of the processors of the processing system are further configured to generate the prompt by combining text-based information and non-text-based information.

14. The computing device according to claim 12, wherein the non-text-based information includes a description based on audio or video sensor data.

15. The computing device according to claim 10, wherein one or more of the processors of the processing system are further configured to select a model size to be used in an information submodule for processing non-text-based information, based on one or more of the determined current user status, the context of the incoming communication, or the user's privacy preferences for the automated reply response.

16. One or more of the processors of the processing system Based on the received user input, a privacy-aware, multimodal automated reply message is generated. The generated privacy-aware multimodal automated reply message is sent to the computing device that initiated the incoming call or message. It is further configured to perform the automatic reply action based on the received user input. The computing device according to claim 10.

17. The computing device according to claim 10, wherein one or more of the processors of the processing system are further configured to determine the user's privacy preferences for the auto-reply response based on information identifying a category of information that the user may allow to be included in the auto-reply response, based on at least one of the context of the incoming call or message, the caller of the incoming call or message, the user's current location, or the user's current activity.

18. The computing device according to claim 10, wherein one or more of the processors of the processing system are further configured to provide a machine learning module with the user input selection of one of the received personal response suggestions, in order to improve the generation of prompts based on selected multimodal information and the user's privacy preferences for the automated response.

19. A computing device, means for collecting multimodal information relating to the user of the computing device, Means for determining the current user status based on the collected multimodal information, A means for determining a user's privacy preferences regarding automated reply responses, A means for generating a prompt based on selected multimodal information and the user's privacy preferences for the automated response, and for inputting the prompt into a Large-Scale Language Model (LLM), Means for receiving a list of personal response proposals from the generated LLM, In response to rendering the received personal response proposals on the electronic display of the computing device, means for receiving a user input selection from among the received personal response proposals, A means for performing an automated reply action based on the received user input, A computing device equipped with [a certain feature].

20. The computing device according to claim 19, further comprising means for activating or deactivating one or more information submodules configured to receive data input and output text suitable for use when prompting the generating LLM, based on the determined current user status and the user's privacy preferences for the determined automated response.

21. The computing device according to claim 20, further comprising means for processing non-text-based information by at least one of the active information submodules in order to generate text suitable for input to the generating LLM.

22. The computing device according to claim 21, wherein the means for generating the prompt comprises means for generating the prompt by combining text-based information and non-text-based information.

23. The computing device according to claim 21, wherein the non-text-based information includes a description based on audio or video sensor data.

24. The computing device according to claim 19, further comprising means for selecting a model size to be used in an information submodule to process non-text-based information based on one or more of the determined current user status, the context of the incoming communication, or the user's privacy preferences for the automated reply response.

25. The means for performing the automatic reply action based on the received user input is: A means for generating a privacy-aware, multimodal, automated reply message based on the received user input, Means for sending the generated privacy-aware multimodal automated reply message to the computing device that initiated the incoming call or message, The computing device according to claim 19, comprising:

26. The computing device according to claim 19, wherein the means for determining the user's privacy preferences for the automated reply response comprises means for determining the user's privacy preferences for the automated reply response based on information identifying a category of information that the user may allow to be included in the automated reply response, based on at least one of the context of the incoming call or message, the caller of the incoming call or message, the user's current location, or the user's current activity.

27. The computing device according to claim 19, further comprising means for providing the user input selection of one of the received personal response suggestions to a machine learning module in order to enable the generation of prompts to be improved based on selected multimodal information and the user's privacy preferences for the automated response.

28. A non-temporary processor-readable medium storing processor-executable instructions configured to cause one or more processors in a computing device's processing system to perform an operation, wherein the operation is Collect multimodal information about the user of the computing device, Based on the collected multimodal information, the current user status is determined, Determining the user's privacy preferences regarding automated reply responses, The process involves generating a prompt based on the selected multimodal information and the user's privacy preferences regarding the automated response, and inputting the prompt into a Large-Scale Language Model (LLM). Receiving a list of personal response proposals from the aforementioned generated LLM, In response to rendering the received personal response proposals on the electronic display of the computing device, the system receives a user input selection from among the received personal response proposals. The system performs an automated reply action based on the user input received, Non-temporary processor-readable media, including [specific data / information].

29. The non-temporary processor-readable medium according to claim 28, wherein the stored processor-executable instructions are configured to cause one or more processors of the processing system of the computing device to perform an operation, the operation further comprising activating or deactivating one or more information submodules configured to receive data input and output text suitable for use when prompting the generating LLM, based on the determined current user status and the user's privacy preferences for the determined auto-response.

30. The non-temporary processor-readable medium according to claim 28, wherein the stored processor-executable instructions are configured to cause one or more processors of the processing system of the computing device to perform an operation, and the operation further comprises providing a machine learning module with the user input selection of one of the received personal response suggestions to enable the generation of prompts based on selected multimodal information and the user's privacy preferences for the automated response.

Images