Situational-context based control of user-assistive actuators using multi-modal neural language (MMNL) model

Abstract

An electronic device and a method for situational-context based control of user-assistive actuators using multi-modal neural language (MMNL) model, is provided. The electronic device monitors first sensor data associated with a first user. The electronic device determines a world-view model of environment corresponding to the first user. The electronic device applies a MMNL model on the determined world-view model and the monitored first sensor data, and determines a situational context associated with the environment and the first user, based on the application of the MMNL model. The electronic device receives a first user-input based on the determined situational context, and further selects, from a set of tasks, a first task based on the received first user-input. The electronic device further controls a first user-assistive actuator associated with the first user to execute at least partially the selected first task.

Description

SITUATIONAL-CONTEXT BASED CONTROL OF USER-ASSISTIVE ACTUATORS USING MULTI-MODAL NEURAL LANGUAGE (MMNL) MODELCROSS-REFERENCE TO RELATED APPLICATIONS / INCORPORATION BY REFERENCE

[0001] This Application also makes reference to Indian Provisional Application No. 202411051084, which was filed on July 03, 2024. The above stated Patent Applications are hereby incorporated herein by reference in their entirety.FIELD

[0002] Various embodiments of the disclosure relate to user-assistive actuators. More specifically, various embodiments of the disclosure relate to situational-context based control of user-assistive actuators using multi-modal neural language (MMNL) model.BACKGROUND

[0003] Robotic assistance is increasingly essential in helping users plan and execute tasks due to its ability to enhance precision, efficiency, and consistency. Robots can handle repetitive and time-consuming tasks with high accuracy, reducing the likelihood of human error and freeing up users to focus on more strategic and creative aspects of their projects. Additionally, robotic systems can operate around the clock, significantly speeding up the execution of tasks and improving overall productivity. Robotic assistance is a boon for mobility disabled individuals, especially those who require full-time caretakers to be present with them to help with daily activities. Robotic assistance may help such individuals in performing daily chores that they are unable to perform themselves.

[0004] However, integration of the robotic assistance also presents new challenges, including potential control errors, interaction issues, and the necessity for comprehensive and real-time training. Further, the robotic assistance may not be efficiently programmedto reliably assist a user in planning and executing a task. The challenge also lies in effectively interaction with and adaption of the robotic assistance to dynamic and unpredictable environments.

[0005] Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of described systems with some aspects of the present disclosure, as set forth in the remainder of the present application and with reference to the drawings.SUMMARY

[0006] An electronic device and a method for situational-context based control of user- assistive actuators using multi-modal neural language (MMNL) model is provided substantially as shown in, and / or described in connection with, at least one of the figures, as set forth more completely in the claims.

[0007] These and other features and advantages of the present disclosure may be appreciated from a review of the following detailed description of the present disclosure, along with the accompanying figures in which like reference numerals refer to like parts throughout.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 is a diagram that illustrates an exemplary network environment for situational-context based control of user-assistive actuators using multi-modal neural language (MMNL) model, in accordance with an embodiment of the disclosure.

[0009] FIG. 2 is a block diagram that illustrates an exemplary electronic device of FIG.1 , in accordance with an embodiment of the disclosure.

[0010] FIG. 3 is a diagram that illustrates an execution pipeline for situational-context based control of user-assistive actuators using multi-modal neural language (MMNL) model, in accordance with an embodiment of the disclosure.

[0011] FIG. 4A and FIG. 4B collectively illustrate a block diagram representing schema for situational-context based control of user-assistive actuators using MMNL model in a multi-user environment, in accordance with an embodiment of the disclosure.

[0012] FIG. 5 is a diagram that illustrates examples of synchronized interaction of users in multi-user environment, in accordance with an embodiment of the disclosure.

[0013] FIG. 6 is a diagram that illustrates an example of situational-context based control of user-assistive actuators, in accordance with an embodiment of the disclosure.

[0014] FIG. 7 is a diagram that illustrates an example of situational-context based control of a user-assistive actuator for execution of sequential tasks, in accordance with an embodiment of the disclosure.

[0015] FIG. 8 is a flowchart that illustrates exemplary operations of a method for situational-context based control of user-assistive actuators using MMNL model, in accordance with an embodiment of the disclosure.DETAILED DESCRIPTION

[0016] The following described implementation may be found in an electronic device and method for situational-context based control of user-assistive actuators using multimodal neural language (MMNL) model. Exemplary aspects of the disclosure may provide an electronic device (for example, a mobile phone, a smart phone, a desktop, a laptop, a personal computer, and the like), which may include circuitry that may monitor, by use of a biometric sensor system, first sensor data associated with a first user. Further, the circuitry may determine a world-view model of an environment corresponding to the firstuser. Further, the circuitry may apply a multi-modal neural language (MMNL) model on the determined world-view model and the monitored first sensor data. Further, based on the application of the MMNL model, the circuitry may determine a situational context associated with the environment and the first user. Further, the circuitry may receive a first user-input based on the determined situational context. Further, the circuitry may select, from a set of tasks, a first task based on the received first user-input. The set of tasks may be associated with an interaction of the first user and the environment. Further, the circuitry may control a first user-assistive actuator associated with the first user to execute at least partially the selected first task.

[0017] Typically, dynamics of human-robot interactions (HRI) involves tackling several complex challenges. Effective communication is crucial, yet it is often hindered by the fundamental differences in language processing between humans and robots. Robots must also be capable of predicting and understanding the intricate and often unpredictable nature of human behavior. Long-term interactions add another layer of complexity, requiring robots to continuously adapt to evolving human behaviors and preferences. Paramount to HRI is the safety of humans, especially in shared environments, along with the protection of their privacy, given the data collected during interactions. The ethical implications of robots’ decisions, particularly in ambiguous situations, are also a significant concern. Moreover, establishing metrics for evaluating HRI is essential for consistent advancement and comparison of systems. Further, the impact of robots on human social structures and psychological health is an area that demands extensive research. These multifaceted issues underscore the necessity for interdisciplinary collaboration, drawing on expertise from fields such as robotics, Al, healthcare, and design, to enhance the integration of robots into daily human life and activities.

[0018] In the healthcare sector, focusing on a limited set of variables and physiological signals in the context of patient care can have significant implications for both patient safety and comfort. When monitoring is restricted to a few parameters, critical changes in a patient’s condition may go unnoticed, potentially leading to adverse events. For instance, relying solely on heart rate and blood pressure without considering other vital signs, like respiratory rate or oxygen saturation, could miss early signs of deterioration. Moreover, the comfort of patients can be affected by the scope of monitoring. Overemphasis on certain signals might lead to unnecessary interventions, causing discomfort or anxiety, while under-monitoring can result in a lack of timely care. It is essential to strike a balance between comprehensive monitoring that ensures safety and selective monitoring that respects patient comfort and avoids alarm fatigue among healthcare providers. While a focused approach to monitoring can be beneficial in certain contexts, it is crucial to consider the broader implications for patient safety and comfort. An integrated approach that accounts for a wide range of physiological signals and environmental factors is key to delivering high-quality patient care. Unlike the most existing methods, the disclosed electronic device provides a heuristic framework, which may be utilized for a broad spectrum of variables, aiming for a thorough grasp of interaction dynamics between users and user-assistive actuators (such as robot, wheelchair, humanoid, etc). Also, it is essential to address these gaps to enhance the safety and efficacy of interaction between users and user-assistive actuators in sensitive settings, like healthcare. The disclosed electronic device provides a symbiotic relationship that ensures that the users to retain control over the task while benefiting from the user-assistive actuators’ precision, efficiency, and ability to handle repetitive or physically demanding activities.

[0019] FIG. 1 is a diagram that illustrates an exemplary network environment for situational-context based control of user-assistive actuators using multi-modal neural language (MMNL) model, in accordance with an embodiment of the disclosure. With reference to FIG. 1 , there is shown a network environment 100. The network environment 100 includes an electronic device 102, a biometric sensor system 104, an environmental sensor system 106, a multi-modal neural language (MMNL) model 108, a set of user- assistive actuators 110, a server 112, a database 114, and a communication network 116. The electronic device 102, the biometric sensor system 104, the environmental sensor system 106, the MMNL model 108, the set of user-assistive actuators 110, and the server 112 may communicate with one another through one or more networks (such as, the communication network 116). The server 112 may be associated with the database 114. The set of user-assistive actuators 110 may include a user-assistive actuator 110-1 , a user-assistive actuator 110-2 ... a user-assistive actuator 110-N. In FIG. 1 , there is further shown a set of users 118 associated with the set of user-assistive actuators 110, where the set of users 118 may include a user 118-1 , a user 118-2 ... a user 118-K. Each user of the set of users 118 may be associated with at least one user-assistive actuator of the set of user-assistive actuators 110. Though fixed number of user-assistive actuators (N user-assistive actuators) and fixed number of users (K users) are illustrated in FIG. 1. However, number of user-assistive actuators and users connected to the electronic device 102 may variate in real-time, which is well within the scope of the disclosure. In some embodiments the set of users 118 may include for example, an operator, a developer, a researcher, a domain expert, a tester, an authorized user, a patient, and the like.

[0020] The electronic device 102 may include suitable logic, circuitry, interfaces, and / or code that may be configured to monitor, by use of the biometric sensor system 104, firstsensor data associated with a first user (for instance, user 118-1 ) of the set of users 118. The electronic device 102 may further determine a world-view model of an environment corresponding to the user 118-1. The electronic device 102 may further apply the MMNL model 108 on the determined world-view model and the monitored first sensor data. Upon application of the MMNL model 108, the electronic device 102 may determine a situational context associated with the environment and the first user 118-1. The electronic device 102 may further receive a first user-input based on the determined situational context. The electronic device 102 may further select, from a set of tasks, a first task based on the received first user-input. The set of tasks may be associated with an interaction of the first user 118-1 and the environment. The electronic device 102 may further control a first user- assistive actuator (for instance, user-assistive actuator 110-1 ) of the set of user-assistive actuators 110 associated with the first user 118-1 to execute at least partially the selected first task. Examples of the electronic device 102 may include, but may not be limited to, a desktop, a tablet, a television (TV), a laptop, a computing device, a smartphone, a cellular phone, a mobile phone, a consumer electronic (CE) device having a display.

[0021] The biometric sensor system 104 may include suitable logic, circuitry, interfaces, and / or code that may be configured to monitor first sensor data (for instance, biometric data) associated with the first user 118-1. The biometric sensor system 104 may include a plurality of sensors including, but not limited to, a brain-computer interface (BCI) sensor, an electroencephalography (EEG) sensor, an electrocardiography (ECG) sensor, an electromyography (EMG) sensor, a facial electromyography (fEMG) sensor, a photoplethysmogram (PPG) sensor, an eye tracker, an electrooculography (EOG) sensor, a voice recognition sensor, a galvanic skin response (GSR) sensor, an electrodermal activity (EDA) sensor, a face detector, a motion sensor, a muscle sensor, a blood pressuresensor, a photoplethysmogram (PPG) sensor, a laser doppler flowmetry (LDF) sensor, an inertial measurement unit (IMU), a temperature sensor, a humidity sensor, a sleep tracker, a magnetometer, and the like. Further, the first sensor data may include data related to, but not limited to, brain, neurons, voluntary / involuntary response, heart dynamics, movement of eyes, voice, skin response, dermal response, facial features, facial expression classification, subvocalization, inner speech, motion, muscles, blood pressure, blood volume changes, blood flow, blood oxygen level, respiration rate, inertia, temperature, humidity, sleep pattern, or magnetic fields around the body of the first user 118-1. At least one sensor of the biometric sensor system 104 may be placed around the first user 118-1 , such that the sensor may monitor the first sensor data associated with the first user 118-1. In one embodiment, the biometric sensor system 104 may transmit the first sensor data directly to the electronic device 102. In another embodiment, remotely located biometric sensor system 104 may transmit the first sensor data to the server 112.

[0022] The environmental sensor system 106 may include suitable logic, circuitry, interfaces, and / or code that may be configured to monitor second sensor data associated with the environment. Further, based on the second sensor data associated with the environment, the world-view model of the environment may be determined. The environmental sensor system 106 may include a multitude of sensors including, but not limited to, a temperature sensor, a humidity sensor, an atmospheric pressure sensor, a sunlight sensor, a noise sensor, an image sensor, or an object tracker. Further, the second sensor data may include data related to temperature, humidity, pressure, solar irradiance, noise, image, objects present in vicinity, type of objects, and the like. At least one sensor of the environmental sensor system 106 may be placed around each user of the set of users 118, such that the sensor may monitor second sensor data associated with theenvironment around respective user. For instance, the temperature sensor and the humidity sensor may be positioned in a room associated with the first user 118-1 , and may monitor the environment of the room. In another instance, the environmental sensor system 106 may be integrated in a wearable device associated with the first user 118-1 , so that the environmental sensor system 106 may continuously monitor environment around the user 118-1 even when the user is in motion. In one embodiment, the environmental sensor system 106 may transmit the second sensor data directly to the electronic device 102. In another embodiment, remotely located environmental sensor system 106 may transmit the second sensor data to the server 112.

[0023] The MMNL model 108 may be a machine learning model that may be configured to determine a situational context associated with the environment and the first user. The MMNL model 108 may also map world-view model of the environment with the first sensor data associated with the first user 118-1. The MMNL model 108 may be trained on a large dataset associated with potential situations to interpret situational context or other types of complex data. In certain instances, the dataset may be particular to scenarios such as gaming or other such multi-user operating platforms.

[0024] In an embodiment, the MMNL model 108 may be a type of an artificial intelligence system (also referred to as an artificial deep neural network) configured to process and understand the first sensor data and the second sensor data. In an example embodiment, the MMNL model 108 may be a large language model, such as a m eta-transform er trained on multi-modal data (biometric data and environment related data). The multi-modal data may correspond to an image, a point cloud, a text, an audio, infrared-range-based data, hyper-spectrum-based data, X-ray based data, time-series related data, tabular data, graph-related data, or inertia-related data. In another example embodiment, the MMNLmodel 108 may be a meta-transformer trained on the multi-modal data (biometric data and environment related data), a transformer-based decoder-only model, an encoder-decoder model (that uses transformers), or a model that uses neural networks other than transformers.

[0025] As an artificial deep neural network, the MMNL model 108 may be referred to as a computational network or a system of artificial neurons in a neural network, arranged in a plurality of layers, as nodes. The plurality of layers of the neural network may include an input layer, one or more hidden layers, and an output layer. Each layer of the plurality of layers may include one or more nodes (or artificial neurons). Outputs of all nodes in the input layer may be coupled to at least one node of hidden layer(s). Similarly, inputs of each hidden layer may be coupled to outputs of at least one node in other layers of the neural network. Outputs of each hidden layer may be coupled to inputs of at least one node in other layers of the neural network. Node(s) in the final layer may receive inputs from at least one hidden layer to output a result. The number of layers and the number of nodes in each layer may be determined from hyper-parameters of the neural network. Such hyper-parameters may be set before or after training the neural network on a training dataset.

[0026] Each node of the MMNL model 108 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters, tunable during training of the network. The set of parameters may include, for example, a weight parameter, a regularization parameter, and the like. Each node may use the mathematical function to compute an output based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the neural network. All or some of the nodes of the neural network may correspond to the same or a different mathematical function.

[0027] In training of the MMNL model 108, one or more parameters of each node of the neural network may be updated based on whether an output of the final layer for a given input (from the training dataset) matches a correct result based on a loss function for the neural network. The above process may be repeated for the same or a different input until a minima of loss function is achieved, and a training error is minimized.

[0028] The MMNL model 108 may include electronic data, which may be implemented as, for example, a software component of an application executable on the electronic device. The MMNL model 108 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device. The MMNL model 108 may include code and routines configured to enable a computing device, such as the electronic device to perform one or more operations such as mapping of the first sensor data and the second sensor data, and generation of the situational context. Additionally, or alternatively, the MMNL model 108 may be implemented using hardware including a processor, a microprocessor (e.g., to perform or control performance of one or more operations), a field- programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). Alternatively, in some embodiments, the MMNL model 108 may be implemented using a combination of hardware and software.

[0029] The set of user-assistive actuators 110 may include user-assistive actuator 110- 1 , user-assistive actuator 110-2 ... user-assistive actuator 110-N. Each user-assistive actuator of the set of user-assistive actuators 110 may include suitable logic, circuitry, interfaces, and / or code that may be configured to execute a task selected by the first user 118-1. Further, the set of user-assistive actuators 110 may include a robot, an autonomous vehicle, a micro-mobility vehicle, a wheelchair, a recliner, a multi-functional bed, a television, a robotic arm, a humanoid, a human-interfaced machine, an industrial machine,or any device including software and hardware for user and the like. Each user-assistive actuator of the set of user-assistive actuators 110 may be trained to operate efficiently using the MMNL model 108.

[0030] The server 112 may be implemented as a cloud server that may be configured to acquire the first sensor data from the biometric sensor system 104, and the second sensor data from the environmental sensor system 106. The server 112 may also store a plurality of tasks, which includes the set of tasks. Further, the first sensor data and the second sensor data may be shared with the electronic device 102. The server 112 may be implemented using on-premises hosting (local servers), colocation hosting (third-party data centers), bare metal servers (dedicated servers), edge computing (local data processing), fog computing (decentralized data processing), mesh computing (distributed computing), hybrid cloud (combination of on-premises and cloud), or multi-cloud (multiple cloud providers).

[0031] The server 112 may execute operations through web applications, cloud applications, HTTP requests, repository operations, file transfer, and the like. Example implementations of the server 112 may include, but are not limited to, a database server, a file server, a web server, an application server, a mainframe server, or a cloud computing server.

[0032] In at least one embodiment, the server 112 may be implemented as a plurality of distributed cloud-based resources by use of several technologies that are well known to those ordinarily skilled in the art. A person with ordinary skill in the art will understand that the scope of the disclosure may not be limited to the implementation of the server 112 and the electronic device 102 as two separate entities. In certain embodiments, the functionalities of the server 112 can be incorporated in its entirety or at least partially in theelectronic device 102, without a departure from the scope of the disclosure. In certain embodiments, the server 112 may host the database 114. Alternatively, the server 112 may be separate from the database 114 and may be communicatively coupled to the database 114.

[0033] The database 114 may be configured to store the plurality of tasks. The database 114 may also store a URL or a path of various world-view models and the plurality of tasks. The database 114 may be derived from data off a relational or non-relational database, or a set of comma-separated values (csv) files in conventional or big-data storage. The database 114 may be stored or cached on a device, such as the server 112 or the electronic device 102. The device storing the database 114 may be configured to receive a command from the electronic device 102 for retrieving world-view models or the plurality of tasks based on a DB query. In response, the device of the database 114 may be configured to retrieve and provide required world-view model or the set of tasks. In some embodiments, the database 114 may be hosted on a plurality of servers stored at same or distinct locations. The operations of the database 114 may be executed using hardware including a processor, a microprocessor (e.g., to perform or control performance of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other instances, the database 114 may be implemented using software.

[0034] The communication network 116 may include a communication medium through which the electronic device 102 may communicate with the biometric sensor system 104, the environmental sensor system 106, the MMNL model 108, the set of user-assistive actuators 110, and the server 112. Examples of the communication network 116 may include, but are not limited to, the Internet, a cloud network, a Wireless Fidelity (Wi-Fi)network, a Personal Area Network (PAN), a Local Area Network (LAN), and / or a Metropolitan Area Network (MAN). Various devices in the network environment 100 may be configured to connect to the communication network 116, in accordance with various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of a Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11 , light fidelity(Li-Fi), 802.16, IEEE 802.11 s, IEEE 802.11 g, multi-hop communication, wireless access point (AP), device to device communication, cellular communication protocols, and / or Bluetooth (BT) communication protocols, or a combination thereof.

[0035] In operation, the electronic device 102 may be configured to monitor first sensor data associated with the user 118-1. The electronic device 102 may be configured to communicate with the biometric sensor system 104 and receive the first sensor data (biometric data) associated with the user 118-1. In an embodiment, the first sensor data may include data related to brain, neurons, voluntary / involuntary response, heart dynamics, movement of eyes, voice, skin response, dermal response, facial features, facial expression classification, subvocalization, inner speech, motion, muscles, blood pressure, blood volume changes, blood flow, blood oxygen level, respiration rate, inertia, temperature, humidity, sleep pattern, magnetic fields, and the like, around the body of the user 118-1. The monitoring of the first sensor data is described further, for example, in FIG. 3 (at 302).

[0036] The electronic device 102 may further be configured to determine a world-view model of an environment corresponding to the user 118-1. The electronic device 102 may be configured to communicate with the environmental sensor system 106 and may receivethe second sensor data associated with the user 118-1. In an embodiment, the second sensor data may include data related to temperature, humidity, pressure, solar irradiance, noise, image, objects present in vicinity, type of objects, and the like. Further, based on the second sensor data associated with the environment, the electronic device 102 may determine the world-view model of the environment. The determination of the world-view model of the environment is described further, for example, in FIG. 3 (at 304).

[0037] The electronic device 102 may further be configured to apply the MMNL model 108 on the determined world-view model and the monitored first sensor data. The electronic device 102 may feed the world-view model and the first sensor data in the MMNL model 108, which may further process the fed world-view model and first sensor data. The application of the MMNL model 108 is described further, for example, in FIG. 3 (at 306).

[0038] Based on the application of the MMNL model 108, the electronic device 102 may further be configured to determine a situational context associated with the environment and the first user 118-1. The electronic device 102 may map, based on the application of the MMNL model 108, the determined world-view model of the environment with the first sensor data associated with the first user 118-1. The determination of the situational context may further be based on the mapping of the determined world-view model with the first sensor data. The determination of the situational context is described further, for example, in FIG. 3 (at 308).

[0039] The electronic device 102 may further be configured to receive a first user-input based on the determined situational context. In one embodiment, the electronic device 102 may receive the first user-input from the first user 118-1 through the biometric sensor system 104 itself. In other embodiment, the electronic device 102 may receive the first user-input from an authenticated user other than the first user 118-1 (for instance, arelative, a caretaker, or a doctor of the first user 118-1 ) through a user interface (for instance, a smart phone, a GUI interface, a tab, and the like) connected to the electronic device 102 through the communication network 116. The reception of the first user-input is described further, for example, in FIG. 3 (at 310).

[0040] Based on the received first user-input, the electronic device 102 may further be configured to select a first task from a set of tasks. In an embodiment, the electronic device 102 may retrieve the set of tasks from a plurality of tasks stored in the server 112 or the database 114, where the set of tasks may be retrieved based an interaction of the first user 118-1 and the environment. Further, the electronic device 102 may select the first task from the retrieved set of tasks. The selection of the first task from the set of tasks is described further, for example, in FIG. 3 (at 312).

[0041] The electronic device 102 may further control first user-assistive actuator 110-1 associated with the first user 118-1 to execute at least partially the selected first task. In one embodiment, the electronic device 102 may control, through the MMNL model 108, actuation of the first user-assistive actuator 110-1 , which may be operated to execute a portion of the first task. In another embodiment, the electronic device 102 may update the MMNL model 108, and may further control the first user-assistive actuator 110-1 to execute whole of the selected first task. In some embodiments, the circuitry 202 may update the MMNL model 108, and may further control the first user-assistive actuator 110-1 to adapt to real-time situations, autonomously perform complex tasks, and provide an accurate response in any given environment.

[0042] The actuation of the first user-assistive actuator 110-1 may assist the first user 118-1 in execution of tasks (for instance, the selected first task) beyond users physical and cognitive limitations of the user 118-1.

[0043] In an embodiment, the circuitry 202 may track movements or actions performed by the first user-assistive actuator 110-1 for the execution of the first task. Further, the circuitry 202 may store data associated with the tracked movements or actions on the server 112 or the memory 204. The stored data may serve as long-term, behavioral and working memory representations for the first user-assistive actuator 110-1. The stored data may also be utilized to understand a level of user disability or a level of preferred user assistance. The controlling of the first user-assistive actuator 110-1 is described further, for example, in FIG. 3 (at 314).

[0044] FIG. 2 is a block diagram that illustrates an exemplary electronic device of FIG.1 , for situational-context based control of user-assistive actuators using MMNL model, in accordance with an embodiment of the disclosure. FIG. 2 is explained in conjunction with elements from FIG. 1 . With reference to FIG. 2, there is shown a block diagram 200 of the electronic device 102. The electronic device 102 may include a circuitry 202, a memory 204, a network interface 206, and an input / output (I / O) device 208. In at least one embodiment, the I / O device 208 may also include an electronic user interface (III), the biometric sensor system 104, and the environmental sensor system 106. In at least one embodiment, the memory 204 may include the MMNL model 108. The circuitry 202 may be communicatively coupled to the memory 204, the network interface 206, and the I / O device 208, through wired or wireless communication of the electronic device 102.

[0045] The circuitry 202 may include suitable logic, circuitry, and interfaces that may be configured to execute program instructions associated with different operations to be executed by the electronic device 102. The operations may include monitoring first sensor data associated with first user 118-1. The operations may further include determination of world-view model of an environment corresponding to the first user 118-1. The operationsmay further include application of the MMNL model 108 on the determined world-view model and the monitored first sensor data. The operations may further include determination of situational context associated with the environment and the first user, based on the application of the MMNL model 108. The operations may further include reception of first user-input based on the determined situational context. The operations may further include selection of first task from the set of tasks based on the received first user-input.

[0046] The circuitry 202 may include one or more specialized processing units, which may be implemented as an integrated processor or a cluster of processors that perform the functions of the one or more specialized processing units, collectively. The circuitry 202 may be implemented based on a number of processor technologies known in the art. Examples of implementations of the circuitry 202 may be an x86-based processor, a Graphics Processing Unit (GPU), a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computing (CISC) processor, a microcontroller, a central processing unit (CPU), and / or other computing circuits.

[0047] The memory 204 may include suitable logic, circuitry, interfaces, and / or code that may be configured to store the program instructions to be executed by the circuitry 202. The program instructions stored on the memory 204 may enable the circuitry 202 to execute operations of the circuitry 202 (and / or the electronic device 102). In at least one embodiment, the memory 204 may store the MMNL model 108. Further, the memory 204 may store path or URL associated with a set of tasks (for instance, set of tasks 210). Examples of implementation of the memory 204 may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), Hard Disk Drive (HDD), a Solid-State Drive (SSD), a CPU cache, and / or a Secure Digital (SD) card.

[0048] The network interface 206 may comprise suitable logic, circuitry, interfaces, and / or code that may be configured to establish a communication between the electronic device 102, the biometric sensor system 104, the environmental sensor system 106, the MMNL model 108, the set of user-assistive actuators 110, and the server 112, and device of the database 114 via the communication network 116. The network interface 206 may be implemented by use of various known technologies to support wired or wireless communication of the electronic device 102, via the communication network 116. The network interface 206 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer.

[0049] The network interface 206 may be configured to communicate via wireless communication with networks, such as the Internet, an Intranet, a wireless network, a cellular telephone network, a wireless local area network (LAN), or a metropolitan area network (MAN). The wireless communication may be configured to use one or more of a plurality of communication standards, protocols and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), wideband code division multiple access (W-CDMA), Long Term Evolution (LTE), 5th Generation (5G) New Radio (NR), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (such as IEEE 802.11 a, IEEE 802.11 b, IEEE 802.11 g or IEEE 802.11 n), voice over Internet Protocol (VoIP), light fidelity (Li-Fi),Worldwide Interoperability for Microwave Access (Wi-MAX), a protocol for email, instant messaging, and a Short Message Service (SMS).

[0050] The I / O device 208 may include suitable logic, circuitry, interfaces, and / or code that may be configured to receive first sensor data and second sensor data and further control first user-assistive actuator 110-1 based on the first sensor data and the second sensor data. In some embodiments, the I / O device 208 may receive the first sensor data from the biometric sensor system 104. In other embodiments, the I / O device 208 may receive the second sensor data from the environmental sensor system 106. The I / O device 208 may include various input and output devices, which may be configured to communicate with the circuitry 202 and other components, such as the network interface 206. Examples of the input devices may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, and / or a microphone. Examples of the output devices may include, but are not limited to, a display (e.g., the display device 208A) and a speaker.

[0051] The display device 208A may comprise suitable logic, circuitry, interfaces, and / or code that may be configured to display the world-view model of the environment and the set of tasks. The display device 208A may be configured to receive first user-input. In such cases the display device 208A may be a touch screen to receive first user-input associated with the situational context. The display device 208A may be realized through several known technologies such as, but not limited to, a Liquid Crystal Display (LCD) display, a Light Emitting Diode (LED) display, a plasma display, and / or an Organic LED (OLED) display technology, and / or other display technologies.

[0052] Modifications, additions, or omissions may be made to the example electronic device 102 without departing from the scope of the present disclosure. For example, insome embodiments, the example electronic device 102 may include any number of other components that may not be explicitly illustrated or described for the sake of brevity.

[0053] FIG. 3 is a diagram that illustrates an execution pipeline for situational-context based control of user-assistive actuators using multi-modal neural language (MMNL) model, in accordance with an embodiment of the disclosure. FIG. 3 is described in conjunction with elements from FIG. 1 and FIG. 2. With reference to FIG. 3, there is shown an execution pipeline 300 that includes operations involved in situational-context based control of the user-assistive actuators 110 using the MMNL model 108. The execution pipeline 300 may include operations 302 to 314 executed by a computing device, such as, the electronic device 102 of FIG. 1 or the circuitry 202 of FIG. 2. The operations 302 to 314 illustrated in the execution pipeline 300 may be performed by any suitable system, apparatus, or device, such as, by the example electronic device 102 of FIG. 1 , or the circuitry 202 of FIG. 2.

[0054] At 302, an operation for first sensor data monitoring may be executed. In an embodiment, the biometric sensor system 104 may monitor first sensor data (for instance, biometric data) associated with the first user 118-1. The biometric sensor system 104 may include multiple sensors including a brain-computer interface (BCI) sensor, an electroencephalography (EEG) sensor, an electrocardiography (ECG) sensor, an eye tracker, a voice recognition sensor, a galvanic skin response (GSR) sensor, an electrodermal activity (EDA) sensor, a face detector, a motion sensor, a muscle sensor, a blood pressure sensor, a photoplethysmogram (PPG) sensor, a laser doppler flowmetry (LDF) sensor, an inertial measurement unit (IMU), a temperature sensor, a humidity sensor, a sleep tracker, a magnetometer, and the like. These sensors may generate the first sensor data, which may include data related to, but not limited to, brain, neurons,voluntary / involuntary response, heart dynamics, movement of eyes, voice, skin response, dermal response, facial features, facial expression classification, subvocalization, inner speech, motion, muscles, blood pressure, blood volume changes, blood flow, blood oxygen level, respiration rate, inertia, temperature, humidity, sleep pattern, or magnetic fields around the body of the first user 118-1. At least one sensor of the biometric sensor system 104 may be placed around the first user 118-1 , such that the sensor may monitor the first sensor data associated with the first user 118-1. In an instance, the biometric sensor system 104 or some of the sensors associated with the biometric sensor system 104 may be attached to a portion of the body of the first user 118-1. In another instance, the biometric sensor system 104 may be integrated in a wearable device, such as, but not limited to, smart watch, smart cloth, ring, boots, associated with the first user 118-1 , which enables the biometric sensor system 104 to continuously monitor the first sensor data, i.e., biometric data of the user 118-1.

[0055] In one embodiment, the circuitry 202 may communicate with the biometric sensor system 104, and the biometric sensor system 104 may transmit the first sensor data (biometric data) associated with the first user 118-1 directly to the circuitry 202 of the electronic device 102. In another embodiment, remotely located biometric sensor system 104 may transmit the first sensor data to the server 112, and further the circuitry 202 may communicate with the server 112 to retrieve the first sensor data from the server 112.

[0056] At 304, an operation for world-view model determination may be executed. In an embodiment, at 304-A, the environmental sensor system 106 may monitor second sensor data associated with the environment corresponding to the first user 118-1. The environmental sensor system 106 may include multiple sensors including a temperature sensor, a humidity sensor, an atmospheric pressure sensor, a sunlight sensor, a noisesensor, an image sensor, an object tracker, and the like. Further, the second sensor data may include data related to temperature, humidity, pressure, solar irradiance, noise, image, objects present in vicinity, type of objects, and the like. At least one sensor of the environmental sensor system 106 may be placed around each user of the set of users 118, such that the sensor may monitor second sensor data associated with the environment around respective user. For instance, the temperature sensor and the humidity sensor may be positioned in a room associated with the first user 118-1 , and may monitor the environment of the room. In another instance, the environmental sensor system 106 may be integrated in a wearable device associated with the first user 118-1 , so that the environmental sensor system 106 may continuously monitor environment around the user 118-1 even when the user is in motion.

[0057] In one embodiment, the circuitry 202 may communicate with the environmental sensor system 106 to receive the second sensor data associated with the environment directly from the environmental sensor system 106. In another embodiment, remotely located environmental sensor system 106 may transmit the second sensor data to the server 112, and further the circuitry 202 may communicate with the server 112, and may retrieve the second sensor data from the server 112.

[0058] The circuitry 202 may further determine the world-view model of the environment, based on the second sensor data. In an instance, the environmental sensor system 106 may be deployed in a room corresponding to the first user 118-1 to monitor light intensity and air circulation in the room. The environmental sensor system 106 may transmit realtime second sensor data to the circuitry 202. The circuitry 202 may further clean, integrate, and analyze the second sensor data. The circuitry 202 may further visualize the second sensor data (using GIS tools) to create dynamic maps showing portions of the roomcorresponding to the first user 118-1 , where air circulation may be less or light intensity may be more or less, and corresponding trends over time.

[0059] In an embodiment, the circuitry 202 may create 2D dynamic maps showing portions of the room. In another embodiment, the circuitry 202 may create 3D dynamic maps showing portions of the room.

[0060] In an embodiment, the circuitry 202 may apply the MMNL model 108 to determine, using spatial intelligence, the world-view model of the environment, and may reason about objects, places, and interactions taking place in the environment in 3D space and time. The circuitry 202 may apply the MMNL model 108 to analyze and reason about 3D world-view model from images and other modalities received by the circuitry 202. The circuitry 202 may further interact with user 118 in real world, using the spatial intelligence. The circuitry 202, through the spatial intelligence, may facilitate the user 118 to reason, move, invent, visualize, or architect virtually anything either physical or virtual. The circuitry 202 may further apply the MMNL model 108 to enable the users 118 to communicate and connect with another, through verbal intelligence.

[0061] At 306, an operation for MMNL model application may be executed. The circuitry 202 may apply the MMNL model 108 on the determined world-view model and the monitored first sensor data. In an embodiment, the circuitry 202 may feed the world-view model and the first sensor data in the MMNL model 108, which may further process the fed world-view model and first sensor data to clean and standardize multimodal data including the world-view model and the first sensor data. The process of cleaning and standardizing involves filtering out noise, handling missing values, and normalizing the data to ensure consistency. Further, the MMNL model 108 may integrate and align the multimodal data. The MMNL model 108 may be trained to understand and correlate therelationships between biometric responses and environmental conditions. This involves using deep learning techniques to create embeddings that represent the different data modalities in a unified space. Further, the MMNL model 108 may execute analysis and interpretation process, which may involve using the (trained) MMNL model 108 to identify patterns and predict outcomes. For example, the MMNL model 108 may analyze how specific environmental conditions influence physiological responses, biometric responses, and sentiments.

[0062] At 308, an operation for situational context determination may be executed. The circuitry 202 may determine a situational context associated with the environment and the first user 118-1. In order to determine the situational context, firstly the circuitry 202 may map the determined world-view model of the environment with the first sensor data associated with the first user 118-1 , based on the application of the MMNL model 108. Further, the circuitry 202 may determine the situational context based on the mapping of the determined world-view model with the first sensor data. Determining the situational context based on the mapping of the determined world-view model with the first sensor data involves a process of aligning the first sensor data and the second sensor data with a pre-established framework or model that may represent the environment or scenario in question. The world-view model may serve as a comprehensive representation of the expected conditions, behaviors, and interactions within a given context. When the first sensor data is received, it may be compared against the world-view model to identify any correlations, deviations, or patterns that may provide insights into the current state of the environment. For instance, in a scenario, the world-view model may include information about room conditions, air circulation patterns, and lighting of the room. The first sensor data, which could include inputs from cameras, BCI, or EEG, may then be mapped ontothe world-view model to monitor physiological or biometric responses of the first user 118- 1 . Based on the mapping, the circuitry 202 may establish a situational context associated with the room and the first user 118-1. This mapping plays a crucial role in ensuring that the circuitry 202 may respond appropriately to real-time situations and maintain safe and efficient operation.

[0063] At 310, an operation for first user-input reception may be executed. The circuitry 202 may receive a first user-input based on the determined situational context. In one embodiment, the electronic device 102 may receive the first user-input from the first user 118-1 through the biometric sensor system 104 itself. For instance, the circuitry 202 may receive biometric response (the first sensor data) from the biometric sensor system 104 as the first user-input from the first user 118-1 , and correspondingly the circuitry 202 may reconfirm the detected biometric response of the first user 118-1. In other embodiment, the electronic device 102 may receive the first user-input from an authenticated user other than the first user 118-1 (for instance, a relative, a caretaker, or a doctor of the first user 118-1 ) through a user interface (for instance, a smart phone, a GUI interface, a tab, and the like ) connected to the electronic device 102 through the communication network 116. The circuitry 202 may receive the first user-input from the authenticated user in order to reconfirm the detected biometric response of the first user 118-1.

[0064] At 312, an operation for task selection may be executed. The circuitry 202 may retrieve a set of tasks from a plurality of tasks stored in the server 112 or the database 114, where the set of tasks may be retrieved based the situational context. In an instance, the set of tasks may include different tasks independent of each other, such as, dimming or brightening of lights, regulating temperature, and enhanced air circulation in a room associated with the first user 118-1. In another instance, the set of tasks may include asequence of interlinked tasks, such as, the set of tasks required to make a comfortable environment for the first user 118-1 to sleep at night time, for example, providing required medicines, providing water, switching off lights, and regulating temperature of a room associated with the first user 118-1.

[0065] Further, the circuitry 202 may select a first task from the set of tasks based on the received first user-input. For instance, in a scenario, the world-view model may include information about room conditions, air circulation patterns, and lighting of the room. The first sensor data, which could include inputs from cameras, BCI, or EEG, may then be mapped onto this model to monitor physiological or biometric responses of the first user 118-1. In an instance, based on the world-view model of the room, a set of tasks including dimming of lights, brightening of lights, enhanced air circulation in room, may be retrieved. Further, based on the first user-input, a first task of the set of tasks may be selected. For instance, dimming of lights and / or enhanced air circulation in room may be selected as the first task. It should be noted that though here the disclosure is talking about selecting only one task (as first task) from the set of tasks, however, more than one task can be selected simultaneously or sequentially from the set of tasks, which is well within the scope of the invention.

[0066] At 314, an operation for first user-assistive actuator controlling may be executed. The circuitry 202 may control first user-assistive actuator 110-1 associated with the first user 118-1 to execute at least partially the selected first task. In an instance, the electronic device 102 may control, through the MMNL model 108, actuation of the first user-assistive actuator 110-1 , which may be operated to execute a portion of the first task.

[0067] In another instance, the circuitry 202 may update the MMNL model 108, and may further control the first user-assistive actuator 110-1 to execute whole of the selected firsttask. In an embodiment, the circuitry 202 may monitor the at least partial execution of the selected first task. The circuitry 202 may further update the MMNL model 108 based on the monitored at least partial execution of the selected first task. Furthermore, the circuitry 202 may control the first user-assistive actuator 110-1 to execute at least partially the selected first task, based on the updated MMNL model 108. In another embodiment, the circuitry 202 may receive a second user-input based on the monitored at least partial execution of the selected first task. In an instance, the second user-input may include execution of a task, instructions, commands, feedback, and the like. The circuitry 202 may further update the MMNL model 108 based on the received second user-input. Furthermore, the circuitry 202 may control the first user-assistive actuator 110-1 to execute at least partially the selected first task, based on the updated MMNL model 108.

[0068] In some embodiments, the circuitry 202 may train and update the MMNL model 108 to perceive, reason, act, adapt, and learn, using techniques including, but not limited to, reinforcement learning, federal learning, Bayesian networks, parallel computing, transfer learning, and self-calibrating Al technique. The reinforcement learning may train the MMNL model 108 to learn to make decisions through trial and error to maximize cumulative rewards. The reinforcement learning may train the MMNL model 108 to learn based on interaction of the first user-assistive actuator 110-1 with an environment and feedback based on actions of the first user-assistive actuator 110-1. The feedback may be in the form of rewards or penalties. The federal learning may enable the first user-assistive actuator 110-1 to learn from distributed data sources whilst ensuring privacy and security. The federal learning may allow distinct machine learning models to collaboratively train shared MMNL model 108 without transferring sensitive data, preserving privacy and reducing reliance on any centralized storage. Alternatively, the circuitry may locally updatemachine learning models and share improvements with the MMNL model 108, preserving both privacy and security. The federal learning may enable sharing of knowledge without compromising privacy or relying on a centralized data source.

[0069] The Bayesian network may refer to a graphical model that represents probabilistic relationships among variables, which may represent environmental sensor data here. The Bayesian network may train the MMNL model 108 to handle uncertain events and make predictions or decisions based on probabilities.

[0070] The parallel computing may train the MMNL model 108 to divide larger tasks into smaller, independent tasks that may be processed simultaneously rather than sequentially. The parallel computing may enable increased computational efficiency, reduced latency, and improved cost efficiency. The parallel computing may enable the circuitry 202 to process inputs from all the sensors of the environmental sensor system 106 simultaneously, to navigate surrounding environment more effectively and efficiently.

[0071] The transfer learning may leverage pre-trained machine learning models to execute new, but similar, tasks. In the transfer learning, the MMNL model 108 trained on one task may be reused and fine-tuned for a related task. For example, in machine vision, fine-tuning of the MMNL model 108 on a small dataset of images may allow the MMNL model 108 to quickly adapt to a specific detection (e.g. cracked surfaces or any other risk.), without any need to train the MMNL model 108 from scratch.

[0072] The self-calibrating Al technique may train and update the MMNL model 108 to autonomously adjust parameters, models, or processes to maintain optimal performance without manual intervention. The self-calibrating Al technique may update the MMNL model 108 to adapt to changes in surrounding environment, hardware, or tasks, ensuring the MMNL model 108 may operate with optimized accuracy and efficiency over time.

[0073] In an embodiment, the circuitry 202 may train and update the MMNL model 108 to assess risks in the execution of the selected first task. The circuitry 202 may further determine actions or movements involved in completion of the selected first task by the first user-assistive actuator 110-1 based on the assessed risk. In another embodiment, the circuitry 202 may train the MMNL model 108 to learn from real-world demonstrations or experiences, and may further update the MMNL model 108 to adapt to new tasks and environments. The circuitry 202 may further control the first user-assistive actuator 110-1 based on the updated MMNL model 108 to execute the selected first task with greater autonomy and precision.

[0074] In some embodiments, the circuitry 202 may update the MMNL model 108, and may further control actuation of the user-assistive actuators 110 to adapt to real-time situations, autonomously perform complex tasks, and provide an accurate response in any given environment. In an instance, for a user-input including a user prompt “Assist me to my room”, the circuitry 202 may apply the MMNL model 108 to process the user prompt. The circuitry 202 may further assist the user 118-1 by guiding through commands such as - i) move left, ii) open the door, iii) move right. The commanded directions may be in cardinal (North, South, East, West) or non-cardinal form (Left, Right, Up, Down). Alternatively, if the user 118-1 is not able to move on his own, the circuitry may control the user-assistive actuator 110-1 , for instance wheelchair on which the user 118-1 is sitting, to execute the task of assistance of the user 118-1 to the room.

[0075] It should be noted that though various embodiments in the disclosure are explained with respect to one user (for instance, user 118-1 ) and one user-assistive actuator (for example, user-assistive actuator 110-1 ), however, the circuitry 202 may efficiently support same operations in a multi-user environment including multiple users(for example, the set of users 118) and a multitude of user-assistive actuators (for example, the set of user-assistive actuators 110) (as elaborated in FIG. 4A and FIG. 4B), which is well within the scope of the disclosure.

[0076] FIG. 4A and FIG. 4B collectively illustrate a block diagram representing schema for situational-context based control of user-assistive actuators using MMNL model in a multi-user environment, in accordance with an embodiment of the disclosure. FIG. 4A and FIG. 4B are described in conjunction with elements from FIG. 1 , FIG. 2, and FIG. 3. With reference to FIG. 4A and FIG. 4B, there is shown a block diagram 400 representing schema for situational-context based control of user-assistive actuators 110 using MMNL model 108. Although illustrated with discrete blocks, the steps and operations associated with one or more of the blocks of the block diagram 400 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

[0077] In a multi-user environment, the circuitry 202 may receive an input 402 that may include the first user data associated with multiple users including user 1 to user N. In an instance, the first user data associated with the input 402 may include input from BCI based sensor, eye tracker, and other such sensors for each user of the multiple users. In an embodiment, the circuitry 202 may receive the input 402 through a user interface (III) layer 404. The III layer 404 acts as a critical component of a software application associated with the circuitry 202, serving as the bridge between the users and the underlying functionality of the circuitry 202. The III layer 404 may encompass all the elements that a user may interact with, for instance, visual components like buttons, menus, icons, and text fields, as well as interactive elements such as touch gestures, voice commands, and keyboard inputs. The III layer 404 may provide an intuitive, efficient, andaesthetically pleasing experience that may enable the users to perform tasks and access features of the electronic device 102 seamlessly. Further, the III layer 404 may be designed to be user-centric, ensuring that the interface is easy to navigate, responsive, and accessible to a diverse range of users. By effectively translating complex backend processes into simple, user-friendly interactions, the III layer 404 may play a pivotal role in enhancing user satisfaction and overall usability of the application.

[0078] The circuitry 202 may further include a control panel 406 that may monitor, manage, and configure various aspects of the electronic device 102. The control panel 406 may include a collection of tools, settings, and controls that provide comprehensive oversight and operational capabilities. In an instance, the control panel 406 may be configured to provide BCI interface 408, conversation control 410, agent pool 414, API 418, working memory 420, and database (or metadata) 422. The control panel 406 may be designed to be user-friendly, featuring organized menus, dashboards, and visual indicators that make it easy to navigate and understand. The control panel 406 may also serve as a command center, enabling users to perform administrative tasks, troubleshoot issues, and customize the functions to meet specific needs. By consolidating these controls into a single interface, the control panel 406 may enhance efficiency, streamline workflows, and ensure that the users have quick access to the critical functions.

[0079] The BCI interface 408 for multiple users may involve simultaneous interpretation and respond to neural signals from the users. This complex process requires advanced signal processing techniques to accurately decode brain activity and translate it into actionable commands or interactions within a shared environment. For instance, in a collaborative virtual reality setting, BCIs may enable multiple users to control avatars, manipulate objects, or communicate through thought alone, enhancing the immersiveexperience. The BCI interface 408 may ensure real-time responsiveness and maintain high accuracy in interpreting diverse neural patterns. Additionally, the BCI interface 408 may involve creating user-friendly interfaces that allow seamless interaction and coordination among the users.

[0080] The conversation control 410 may involve using neural activity to manage and direct the flow of dialogues or conversation in a communication setting. This innovative approach leverages the ability of BCIs to interpret brain signals and translate them into commands or actions, enabling corresponding users to control various aspects of a conversation without the need for physical input. For example, individuals with speech or motor impairments could use BCIs to select pre-defined phrases, modulate their tone, or signal turn-taking in a discussion. The conversation control 410 may be executed based on behavior data retrieved from behavioral knowledge store 412, which may lead to accurate decoding of specific neural patterns associated with different conversational intents, such as wanting to speak, agreeing, or asking a question. This requires sophisticated algorithms capable of real-time processing and high precision to ensure smooth and natural interactions. By facilitating more inclusive and accessible communication, the (BCI-based) conversation control 410 may enhance social interaction and improving the quality of life for individuals with communication challenges.

[0081] The agent pool 414 may include a collection of software agents designed to manage and optimize the processing and interpretation of neural signals from multiple BCI users or devices. In an instance, as illustrated in FIG. 4B, the agent pool 414 may include, but not limited to, at least one of: (i) decision agent 414-1 that may aid in coming to a decision based on the processing and interpretation of neural signals; (ii) a virtual reality (VR) agent 414-2 that may facilitate VR representation of the environment and interactionof the users in the VR environment; (iii) a goal modelling agent 414-3 that may model goals for the users; (iv) a generative artificial intelligence (Gen Al) agent 414-4 that may aid in knowledge generation, dialogue generation, memory generation, and other such operations; and (v) other agents 414-5 that may facilitate in miscellaneous functions to handle distinct tasks. The agents may work collaboratively to handle various tasks such as human-computer interaction, signal acquisition, preprocessing (e.g., noise reduction and artifact removal), feature extraction, interpretation of user intentions by analyzing gestures and emotional cues, and classification of brain activity into meaningful commands or actions. By distributing different tasks across multiple agents, the circuitry 202 may achieve higher efficiency, scalability, and robustness, especially in environments where multiple users are interacting simultaneously or where real-time processing is critical. The agent pool 414 may dynamically allocate resources based on the current workload, ensuring that each BCI device receives the necessary computational power and attention to maintain accurate and responsive performance. Hence, the agent pool 414 may enhance the overall functionality and user experience of BCI applications, enabling more complex and interactive scenarios such as multi-user virtual reality environments, collaborative neurofeedback sessions, or advanced assistive technologies.

[0082] Further, output 416 generated from the agent pool 414 may include a set of tasks 416-1 and memories 416-2. The output 416 may be stored in form of metadata in the database 422 of the control panel 406. The output 416 may also be stored in working memory 420.

[0083] The API 418 may serve as a crucial bridge between BCI hardware and software applications, enabling developers to create programs that can interpret and utilize neural data. The API 418 may provide standardized methods and protocols for accessing,processing, and transmitting brain signals captured by BCIs and associated devices. The API 418 may include functions for signal acquisition, real-time data streaming, signal filtering, feature extraction, and classification of neural patterns. By offering these tools, the API 418 may allow developers to build applications that can respond to brain activity, such as controlling a cursor, operating a prosthetic limb, or navigating a virtual environment. The API 418 may also ensure compatibility and interoperability between distinct BCI devices and software platforms associated with different users, facilitating the integration of BCI capabilities into a wide range of applications.

[0084] Further, the circuitry 202 may execute external task mapping 424 based on the set of tasks 416-1 and the memories 416-2. A first task may be selected from the set of tasks 416-1 based on the external task mapping 424. Further, the circuitry 202 may determine an extent of task (426-1 ) executed by the set of user-assistive actuators 110. The circuitry 202 may further feed the extent of task (426-1 ) executed by the set of user- assistive actuators 110 in sensing module 428. The circuitry 202 may also determine an extent of task (426-2) executed by each user of the set of users 118. The circuitry 202 may further feed the extent of task (426-2) executed by the set of users 118 in the sensing module 428. Further, the sensing module 428 may feed data related to the extent of task (426-1 and 426-2) in the MMNL model 108, which may generate a large world model 430 based on the fed data. Furthermore, the circuitry 202 may include an embedding model 432 that may generate embeddings required for the generation of the large world model 430. In an instance, the embedding model 432 may generate embeddings based on data retrieved from an external knowledge source 434 (whose path may be stored in the server 112).

[0085] In an embodiment, the circuitry 202 may monitor, by use of the biometric sensor system 104, third sensor data associated with a second user. The circuitry 202 may further compare the first sensor data with the third sensor data. The circuitry 202 may further determine an interaction between the first user 118-1 and the second user 118-2, based on the comparison. The circuitry 202 may also control a user-assistive actuator (for example, the user-assistive actuator 110-2 of FIG. 1 ) to execute at least partially the selected first task, based on the determined interaction between the first user 118-1 and the second user 118-2.

[0086] The circuitry 202 may further receive a third user-input based on the determined interaction between the first user and the second user. The circuitry 202 may further select, from the set of tasks, a second task based on the received third user-input. The set of tasks may further be associated with an interaction of the second user 118-2 and the environment. The circuitry 202 may further control a user-assistive actuator (for example, the user-assistive actuator 110-N of FIG. 1 ) associated with the second user 118-2 to execute at least partially the selected second task.

[0087] FIG. 5 is a diagram that illustrates examples of synchronized interaction of users in multi-user environment, in accordance with an embodiment of the disclosure. FIG. 5 is described in conjunction with elements from FIG. 1 , FIG. 2, FIG. 3, FIG. 4A and FIG. 4B. With reference to FIG. 5, there is shown a diagram 500 representing examples of synchronized interaction of users in multi-user environment.

[0088] In an instance, the multi-user environment may include three users - user 1 (represented by 1 ), user 2 (represented by 2), and user 3 (represented by 3). In scenario 502, all the three users may be involved in three separate environments, hence in corresponding graph wavelengths associated with neuronal activities of the users aredistinct. In scenario 504, two users (2 and 3) of the three users may be involved in the same environment, hence in corresponding graph wavelengths associated with neuronal activities of the users 2 and 3 are of similar value, however as user 1 is not interacting with the users 2 and 3, wavelength associated with neuronal activities of the user 1 is distinct.

[0089] FIG. 6 is a diagram that illustrates an example of situational-context based control of user-assistive actuators, in accordance with an embodiment of the disclosure. FIG. 6 is described in conjunction with elements from FIG. 1 , FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, and FIG. 5. With reference to FIG. 6, there is shown a diagram 600 representing an example of situational-context based control of the user-assistive actuators 110.

[0090] The diagram 600 shows user 118-1 wearing a (BCI-based) headset 602. BCI sensors associated with the headset 602 may detect electrical activities in brain of the user 118-1 , and may generate corresponding first data. The first data may then be processed by the circuitry to generate a graph of EEG 604. The generated EEG 604 may further be fed to a BCI-device gateway 606. In one embodiment, the BCI-device gateway 606 may generate a command to control actuation of a first user-assistive actuator 110-1 (for instance, a robot 608) to execute a selected task 610 (for instance, placing an object on a table of sofa set). In an instance, example commands may include a set of instructions including - Up, Down, Right, Left, Stop / Pause, Land-Emergency stop, Land, Takeoff, Take a picture, Move backward, Move forward, and Start video stream.

[0091] In another embodiment, the BCI-device gateway 606 may generate a command to control actuation of another user-assistive actuator (for instance, a wheelchair 612 on which the user 118-1 is sitting). The circuitry 202 may communicate with a BCI / wheelchair interface for actuation of the wheelchair 612. Upon actuation the wheelchair 612 may move in area 614 from one room to another and may reach to the room in which the sofa set isplaced. Further, the wheelchair 612 may adjust height to aid the user 118-1 in accomplishing the selected task 610, i.e. placement of an object on the table of the sofa set.

[0092] Though FIG. 6 illustrates that the user 118-1 is wearing the (BCI-based) headset 602, which includes BCI sensors. However, a range of devices including non-invasive wearables to invasive surgically implanted chips including the BCI sensors may be used for detection of electrical activities in the brain of the user 118-1 , which is well within the scope of the disclosure.

[0093] In some embodiments, the non-invasive wearables such as, but not limited to, headset and headband, may utilize Functional Near-Infrared Spectroscopy (fNIRS) technique to detect electrical activities in the brain of the user 118-1. The fNIRS technique refers to a non-invasive optical imaging technique that may determine electrical activities in the brain of the user 118-1 based on a detection of changes in blood oxygenation levels. In the fNIRS technique, near-infrared light may be emitted into the brain, which may further be absorbed and scattered by tissues of the brain. Further, the circuitry 202 may determine the electrical activities in the brain (in form of neural signals) by monitoring variations in amount of the near-infrared light absorbed by oxygenated and deoxygenated hemoglobin in real-time.

[0094] In an instance, the circuitry 202, through the fNIRS technique, may capture neural signals associated with cognitive tasks. The circuitry 202 may further process and translate the captured neural signals into commands that may control the user-assistive actuators 110, such as the robot 608, the wheelchair 612, and the like. fNIRS technique - based BCI sensors may enable users with motor impairments to communicate with the the user-assistive actuators 110 without requiring any physical movement.

[0095] In some embodiments, in case of invasive BCI wearables, BCI electrodes may be directly implanted in the tissues of the brain of the user 118-1. Close proximity of the BCI electrodes with the tissues may result in detection of precise and high-resolution neural signals. The BCI electrodes may be selected from any of microelectrode array (MEA), electrocorticography (ECoG) electrodes, stereo-electroencephalography (sEEG) electrodes, and deep brain stimulation (DBS) - based electrodes, Local Field Potential (LFP) - based electrodes, single / m ulti-unit recordings, neural dust, intravascular electrode systems, and the like.

[0096] Further, in order to detect neural signals, the BCI-based devices may utilize techniques such as P300 BCI, Motor Imagery (Ml), Steady-State Visual Evoked Potential (SSVEP), Cortical Evoked Potential (CEP), Cognitive state monitoring, and the like. The P300 BCI technique may refer to an event-related potential (ERP) that appears approximately 300 milliseconds after a stimulus. The P300 BCI may be used to detect when the user 118-1 recognizes a specific target among a series of stimuli. The Ml technique may involve imagining a specific movement without actually performing the movement. The Ml technique may generate distinct EEG patterns, which may be used to control the user-assistive actuators 110. The SSVEP technique may be used to detect brain responses to visual stimuli that flicker at a constant frequency. The CEP technique may be used for sensory processing based on detection of brain responses to sensory stimuli, such as visual or auditory signals. The cognitive state monitoring technique may involve using EEG to assess mental state of the user 118-1 by analyzing brainwave patterns based on the detected neural signals.

[0097] The circuitry 202 may receive the detected neural signals from the BCI devices. The circuitry 202 may further process the received neural signals to determine cognitivestates of the user 118-1 , such as stress, relaxation, engagement, cognitive load, and the like.

[0098] The circuitry 202 may further generate the first data by processing the detected neural signals. Further, the circuitry 202 may generate a graph of EEG 604 based on the first data. The generated graph of EEG 604 may be used for image classification and reconstruction. In an instance, the circuitry 202 may analyze the graph of EEG 604 to determine brainwave patterns associated with the neural signals. The circuitry 202 may further identify and recreate visual images perceived by the user 118-1 based on the determined brainwave patterns. The circuitry 202 may analyze the graph of EEG 604 in real-time, while the user 118-1 views or imagines specific images. The circuitry 202 may further apply machine learning models, such as MMNL model 108, on the graph of the EEG 604 to classify corresponding neural signals, and identify distinct patterns associated with different visual stimuli. The image classification may be used to select images or commands based on the brainwave patterns associated with the user 118-1.

[0099] Further, in the process of image reconstruction, the circuitry 202 may recreate actual images based on the graph of the EEG 604. The circuitry 202, utilizing sophisticated computational models, may decode visual processing based on the brainwave patterns of the user 118-1. The circuitry 202 may further generate a visual representation of what the user 118-1 is seeing or imagining, based on the decoded visual processing. The process of generation of the visual representation may enable users with disabilities to communicate or interact with surrounding environment through thought alone. The combination of image classification and reconstruction may leverage significant advancement in visual processing capabilities of the brain.

[0100] FIG. 7 is a diagram that illustrates an example of situational-context based control of a user-assistive actuator for execution of sequential tasks, in accordance with an embodiment of the disclosure. FIG. 7 is described in conjunction with elements from FIG. 1 , FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, FIG. 5, and FIG. 6. With reference to FIG. 7, there is shown a diagram 700 representing an example of situational-context based control of user-assistive actuator.

[0101] The diagram 700 represents an example of situational-context based control of a user-assistive actuator (for instance, wheelchair 702 in which the user 118-1 is sitting) for execution of sequential tasks. The user 118-1 may provide first sensor data through a user-interface 704. In an instance, the user 118-1 may select, on the user-interface 704, a set of tasks including “switch off TV”, “go to dinner table”, “take medicine”, “retire to bed”, and the like. Based on selection of the set of tasks, the circuitry 202 may first enable switching off the TV in the room of the user 118-1. The circuitry 202 may further control actuation and movement of the wheelchair 702 to the dinning table. The circuitry 202 may further facilitate movement of the wheelchair 702 to a medicine container, in order to aid the user 118-1 in taking the medicine. The circuitry 202 may further facilitate movement of the wheelchair 702 to a bed to enable the user 118-1 to retire to the bed.

[0102] FIG. 8 is a flowchart that illustrates exemplary operations of a method for situational-context based control of user-assistive actuators using MMNL model, in accordance with an embodiment of the disclosure. FIG. 8 is explained in conjunction with elements from FIG. 1 , FIG. 2, FIG. 3, FIG. 4A, FIG. 4B, FIG. 5, FIG. 6, and FIG. 7. With reference to FIG. 8, there is shown a flowchart 800. The flowchart 800 may include operations from 802 to 816 and may be implemented by the electronic device 102 of FIG.1 or the circuitry 202 of FIG. 2. The flowchart 800 may start at 802 and proceed to 804.

[0103] At 804, first sensor data associated with first user 118-1 may be monitored, monitor first sensor data associated with first user 118-1. The circuitry 202 may be configured to communicate with the biometric sensor system 104 and receive the first sensor data (biometric data) associated with the first user 118-1. In an embodiment, the first sensor data may include data related to brain, neurons, voluntary / involuntary response, heart dynamics, movement of eyes, voice, skin response, dermal response, facial features, facial expression classification, subvocalization, inner speech, motion, muscles, blood pressure, blood volume changes, blood flow, blood oxygen level, respiration rate, inertia, temperature, humidity, sleep pattern, magnetic fields, and the like, around the body of the first user 118-1 .The monitoring of the first sensor data is described further, for example, in FIG. 3 (at 302).

[0104] At 806, a world-view model of an environment corresponding to the first user 118- 1 may be determined. The circuitry 202 may be configured to communicate with the environmental sensor system 106 and receive the second sensor data associated with the first user 118-1. In an embodiment, the second sensor data may include data related to temperature, humidity, pressure, solar irradiance, noise, image, objects present in vicinity, type of objects, and the like. Further, based on the second sensor data associated with the environment, the electronic device 102 may determine the world-view model of the environment. The determination of the world-view model of the environment is described further, for example, in FIG. 3 (at 304).

[0105] At 808, the MMNL model 108 may be applied. The circuitry 202 may apply the MMNL model 108 on the determined world-view model and the monitored first sensor data. The circuitry 202 may feed the world-view model and the first sensor data in the MMNLmodel 108, which may further process the fed world-view model and first sensor data. The application of the MMNL model 108 is described further, for example, in FIG. 3 (at 306).

[0106] At 810, a situational context associated with the environment and the first user 118-1 may be determined. The circuitry 202 may determine a situational context associated with the environment and the first user 118-1. The circuitry 202 may map, based on the application of the MMNL model 108, the determined world-view model of the environment with the first sensor data associated with the first user 118-1. The determination of the situational context may further be based on the mapping of the determined world-view model with the first sensor data. The determination of the situational context is described further, for example, in FIG. 3 (at 308).

[0107] At 812, a first user-input may be received. The circuitry 202 may receive a first user-input based on the determined situational context. In one embodiment, the circuitry 202 may receive the first user-input from the first user 118-1 through the biometric sensor system 104 itself. In other embodiment, the circuitry 202 may receive the first user-input from an authenticated user other than the first user 118-1 (for instance, a relative, a caretaker, or a doctor of the first user 118-1 ) through a user interface (for instance, a smart phone, a GUI interface, a tab, and the like) connected to the circuitry 202 through the communication network 116. The reception of the first user-input is described further, for example, in FIG. 3 (at 310).

[0108] At 814, a first task from a set of tasks may be selected. The circuitry 202 may retrieve the set of tasks from a plurality of tasks stored in the server 112 or the database 114, where the set of tasks may be retrieved based the situational context. Further, the circuitry 202 may select a first task from the retrieved set of tasks. The selection of the first task from the set of tasks is described further, for example, in FIG. 3 (at 312).

[0109] At 816, first user-assistive actuator 110-1 associated with the first user 118-1 may be controlled. The circuitry 202 may control the first user-assistive actuator 110-1 associated with the first user 118-1 to execute at least partially the selected first task. In one embodiment, the circuitry 202 may control, through the MMNL model 108, actuation of the first user-assistive actuator 110-1 , which may be operated to execute a portion of the first task. In another embodiment, the circuitry 202 may update the MMNL model 108, and may further control the first user-assistive actuator 110-1 to execute whole of the selected first task. In an embodiment, the circuitry 202 may track movements or actions performed by the first user-assistive actuator 110-1 for the execution of the first task. Further, the circuitry 202 may store data associated with the tracked movements or actions on the server 112 or the memory 204. The stored data may serve as long-term, behavioral and working memory representations for the first user-assistive actuator 110-1. The stored data may also be utilized to understand a level of user disability or a level of preferred user assistance. The controlling of the first user-assistive actuator 110-1 is described further, for example, in FIG. 3 (at 314). Control may pass to end.

[0110] Although the flowchart 800 is illustrated as discrete operations, such as 802, 804, 806, 808, 810, 812, 814, and 816 the disclosure is not so limited. Accordingly, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the implementation without detracting from the essence of the disclosed embodiments.

[0111] Various embodiments of the disclosure may provide a non-transitory computer- readable medium and / or storage medium having stored thereon, computer-executable instructions executable by a machine and / or a computer to operate an electronic device (for example, the electronic device 102 of FIG. 1 ). Such instructions may cause theelectronic device 102 to perform operations that may include monitoring, by use of a biometric sensor system (for example, the biometric sensor system 104 of FIG. 1 ), first sensor data associated with a first user (for example, the first user 118-1 of FIG. 1 ). The operations may further include determination of a world-view model of an environment corresponding to the first user 118-1. The operations may further include application of a MMNL model (for example, the MMNL model 108 of FIG. 1 ) on the determined world-view model and the monitored first sensor data. The operations may further include determination of a situational context associated with the environment and the first user, based on the application of the MMNL model 108. The operations may further include reception of a first user-input based on the determined situational context. The operations may further include selection, from a set of tasks, of a first task based on the received first user-input. The set of tasks may be associated with an interaction of the first user and the environment. The operations may further include controlling a first user-assistive actuator (for example, the first user-assistive actuator 110-1 of FIG. 1 ) associated with the first user 118-1 to execute at least partially the selected first task.

[0112] Exemplary aspects of the disclosure may provide an electronic device (such as the electronic device 102 of FIG. 1 ) that includes circuitry (such as the circuitry 202). The circuitry 202 may be configured to monitor, by use of a biometric sensor system (for example, the biometric sensor system 104 of FIG. 1 ), first sensor data associated with a first user (for example, the first user 118-1 of FIG. 1 ). Further, the circuitry 202 may be configured to determine a world-view model of an environment corresponding to the first user 118-1. Further, the circuitry 202 may be configured to apply a MMNL model (for example, the MMNL model 108 of FIG. 1 ) on the determined world-view model and the monitored first sensor data. Further, the circuitry 202 may be configured to determine asituational context associated with the environment and the first user, based on the application of the MMNL model 108. Further, the circuitry 202 may be configured to receive a first user-input based on the determined situational context. Further, the circuitry 202 may be configured to select, from a set of tasks, of a first task based on the received first user-input. The set of tasks may be associated with an interaction of the first user and the environment. Further, the circuitry 202 may be configured to control a first user-assistive actuator (for example, the first user-assistive actuator 110-1 of FIG. 1 ) associated with the first user 118-1 to execute at least partially the selected first task.

[0113] In an embodiment, the biometric sensor system 104 may include at least one of: a brain-computer interface (BCI) sensor, an electroencephalography (EEG) sensor, an electrocardiography (ECG) sensor, an eye tracker, a voice recognition sensor, a galvanic skin response (GSR) sensor, an electrodermal activity (EDA) sensor, a face detector, a motion sensor, a muscle sensor, a blood pressure sensor, a photoplethysmogram (PPG) sensor, a laser doppler flowmetry (LDF) sensor, an inertial measurement unit (IMU), a temperature sensor, a humidity sensor, a sleep tracker, or a magnetometer.

[0114] In an embodiment, the circuitry 202 may further be configured to monitor, by use of an environmental sensor system (for example, the environmental sensor system 106 of FIG. 1 ), second sensor data associated with the environment. The determination of the world-view model may be based on at least the second sensor data associated with the environment.

[0115] In an embodiment, the environmental sensor system 106 may include at least one of: a temperature sensor, a humidity sensor, an atmospheric pressure sensor, a sunlight sensor, a noise sensor, an image sensor, or an object tracker.

[0116] In an embodiment, the circuitry 202 may further be configured to map, based on the application of the MMNL model 108, the determined world-view model of the environment with the first sensor data associated with the first user 118-1. The determination of the situational context may further be based on the mapping of the determined world-view model with the first sensor data.

[0117] In an embodiment, the circuitry 202 may further be configured to determine the set of tasks based on the situational context.

[0118] In an embodiment, the set of tasks may correspond to a sequence of interlinked tasks.

[0119] In an embodiment, the first user-assistive actuator 110-1 may be at least one of: a robot, an autonomous vehicle, a micro-mobility vehicle, a wheelchair, a recliner, a multifunctional bed, a robotic arm, a humanoid, a human-interfaced machine, or a television.

[0120] In an embodiment, the MMNL model 108 may correspond to a meta-transformer trained on multi-modal data.

[0121] In an embodiment, the multi-modal data corresponds to at least one of: an image, a point cloud, a text, an audio, infrared-range-based data, hyper-spectrum-based data, X- ray based data, time-series related data, tabular data, graph-related data, or inertia-related data.

[0122] In an embodiment, the circuitry 202 may further be configured to monitor the at least partial execution of the selected first task. The circuitry 202 may further be configured to update the MMNL model 108 based on the monitored at least partial execution of the selected first task. The circuitry 202 may further be configured to control the first user- assistive actuator 110-1 to execute at least partially the selected first task, based on the updated MMNL model 108.

[0123] In an embodiment, the circuitry 202 may further be configured to monitor the at least partial execution of the selected first task. The circuitry 202 may further be configured to receive a second user-input based on the monitored at least partial execution of the selected first task. The circuitry 202 may further be configured to update the MMNL model 108 based on the received second user-input. The circuitry 202 may further be configured to control the first user-assistive actuator 110-1 to execute at least partially the selected first task, based on the updated MMNL model 108.

[0124] In an embodiment, the circuitry 202 may further be configured to monitor, by use of the biometric sensor system 104, third sensor data associated with a second user (for example, the second user 118-2 of FIG. 1 ). The circuitry 202 may further be configured to compare the first sensor data with the third sensor data. The circuitry 202 may further be configured to determine an interaction between the first user 118-1 and the second user 118-2, based on the comparison.

[0125] In an embodiment, the circuitry 202 may further be configured to control a second user-assistive actuator (for example, the second user-assistive actuator 110-2 of FIG. 1 ) to execute at least partially the selected first task, based on the determined interaction between the first user 118-1 and the second user 118-2.

[0126] In an embodiment, the circuitry 202 may further be configured to receive a third user-input based on the determined interaction between the first user 118-1 and the second user 118-2. The circuitry 202 may further be configured to select, from the set of tasks, a second task based on the received third user-input. The set of tasks may further be associated with an interaction of the second user 118-2 and the environment. The circuitry 202 may further be configured to control a third user-assistive actuator (forexample, the user-assistive actuator 110-N of FIG. 1 ) associated with the second user 118-2 to execute at least partially the selected second task.

[0127] The present disclosure may be realized in hardware, or a combination of hardware and software. The present disclosure may be realized in a centralized fashion, in at least one computer system, or in a distributed fashion, where different elements may be spread across several interconnected computer systems. A computer system or other apparatus adapted for carrying out the methods described herein may be suited. A combination of hardware and software may be a general-purpose computer system with a computer program that, when loaded and executed, may control the computer system such that it carries out the methods described herein. The present disclosure may be realized in hardware that includes a portion of an integrated circuit that also performs other functions. It may be understood that, depending on the embodiment, some of the steps described above may be eliminated, while other additional steps may be added, and the sequence of steps may be changed.

[0128] The present disclosure may also be embedded in a computer program product, which includes all the features that enable the implementation of the methods described herein, and which when loaded in a computer system is able to carry out these methods. Computer program, in the present context, means any expression, in any language, code or notation, of a set of instructions intended to cause a system with an information processing capability to perform a particular function either directly, or after either or both of the following: a) conversion to another language, code or notation; b) reproduction in a different material form. While the present disclosure has been described with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made, and equivalents may be substituted without departing from the scope of thepresent disclosure. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present disclosure without departing from its scope. Therefore, it is intended that the present disclosure is not limited to the particular embodiment disclosed, but that the present disclosure will include all embodiments that fall within the scope of the appended claims.

Claims

CLAIMSWhat is claimed is:1 . An electronic device, comprising: circuitry configured to: monitor, by use of a biometric sensor system, first sensor data associated with a first user; determine a world-view model of an environment corresponding to the first user; apply a multi-modal neural language (MMNL) model on the determined world-view model and the monitored first sensor data; determine a situational context associated with the environment and the first user, based on the application of the MMNL model; receive a first user-input based on the determined situational context; select, from a set of tasks, a first task based on the received first userinput, wherein the set of tasks is associated with an interaction of the first user and the environment; and control a first user-assistive actuator associated with the first user to execute at least partially the selected first task.

2. The electronic device according to claim 1 , wherein the biometric sensor system comprises at least one of: a brain-computer interface (BCI) sensor, an electroencephalography (EEG) sensor,an electrocardiography (ECG) sensor, an electromyography (EMG) sensor, a facial electromyography (fEMG) sensor, a photoplethysmogram (PPG) sensor, an eye tracker, an electrooculography (EOG) sensor, a voice recognition sensor, a galvanic skin response (GSR) sensor, an electrodermal activity (EDA) sensor, a face detector, a motion sensor, a muscle sensor, a blood pressure sensor, a photoplethysmogram (PPG) sensor, a laser doppler flowmetry (LDF) sensor, an inertial measurement unit (IMU), a temperature sensor, a humidity sensor, a sleep tracker, or a magnetometer.

3. The electronic device according to claim 1 , wherein the circuitry is further configured to:monitor, by use of an environmental sensor system, second sensor data associated with the environment, wherein the determination of the world-view model is based on at least the second sensor data associated with the environment.

4. The electronic device according to claim 3, wherein the environmental sensor system comprises at least one of: a temperature sensor, a humidity sensor, an atmospheric pressure sensor, a sunlight sensor, a noise sensor, an image sensor, or an object tracker.

5. The electronic device according to claim 1 , wherein the circuitry is further configured to: map, based on the application of the MMNL model, the determined world-view model of the environment with the first sensor data associated with the first user, wherein the determination of the situational context is further based on the mapping of the determined world-view model with the first sensor data.

6. The electronic device according to claim 1 , wherein the circuitry is further configured to determine the set of tasks based on the situational context.

7. The electronic device according to claim 1 , wherein the set of tasks corresponds to a sequence of interlinked tasks.

8. The electronic device according to claim 1 , wherein the first user-assistive actuator is at least one of: a robot, an autonomous vehicle, a micro-mobility vehicle, a wheelchair, a recliner, a multi-functional bed, a robotic arm, a humanoid, a human-interfaced machine, or a television.

9. The electronic device according to claim 1 , wherein the MMNL model corresponds to a meta-transformer trained on multi-modal data.

10. The electronic device according to claim 9, wherein the multi-modal data corresponds to at least one of:an image, a point cloud, a text, an audio, infrared-range-based data, hyper-spectrum-based data,X-ray based data, time-series related data, tabular data, graph-related data, or inertia-related data.

11. The electronic device according to claim 1 , wherein the circuitry is further configured to: monitor the at least partial execution of the selected first task; update the MMNL model based on the monitored at least partial execution of the selected first task; and control the first user-assistive actuator to execute at least partially the selected first task, based on the updated MMNL model.

12. The electronic device according to claim 1 , wherein the circuitry is further configured to: monitor the at least partial execution of the selected first task;receive a second user-input based on the monitored at least partial execution of the selected first task; update the MMNL model based on the received second user-input; and control the first user-assistive actuator to execute at least partially the selected first task, based on the updated MMNL model.

13. The electronic device according to claim 1 , wherein the circuitry is further configured to: monitor, by use of the biometric sensor system, third sensor data associated with a second user; compare the first sensor data with the third sensor data; and determine an interaction between the first user and the second user, based on the comparison.

14. The electronic device according to claim 13, wherein the circuitry is further configured to control a second user-assistive actuator to execute at least partially the selected first task, based on the determined interaction between the first user and the second user.

15. The electronic device according to claim 13, wherein the circuitry is further configured to: receive a third user-input based on the determined interaction between the first user and the second user; select, from the set of tasks, a second task based on the received third userinput, whereinthe set of tasks is further associated with an interaction of the second user and the environment; and control a third user-assistive actuator associated with the second user to execute at least partially the selected second task.

16. A method, comprising: in an electronic device: monitoring, by use of a biometric sensor system, first sensor data associated with a first user; determining a world-view model of an environment corresponding to the first user; applying a multi-modal neural-language (MMNL) model on the determined world-view model and the monitored first sensor data; determining a situational context associated with the environment and the first user, based on the application of the MMNL model; receiving a first user-input based on the determined situational context; selecting, from a set of tasks, a first task based on the received first userinput, wherein the set of tasks is associated with an interaction of the first user and the environment; and controlling a first user-assistive actuator associated with the first user to execute at least partially the selected first task.

17. The method according to claim 16, further comprising:monitoring, by use of the biometric sensor system, third sensor data associated with a second user; comparing the first sensor data with the third sensor data; and determining an interaction between the first user and the second user, based on the comparison.

18. The method according to claim 17, further comprising controlling a second user- assistive actuator to execute at least partially the selected first task, based on the determined interaction between the first user and the second user.

19. The method according to claim 17, further comprising: receiving a third user-input based on the determined interaction between the first user and the second user; selecting, from the set of tasks, a second task based on the received third user-input, wherein the set of tasks is further associated with an interaction of the second user and the environment; and controlling a third user-assistive actuator associated with the second user to execute at least partially the selected second task.

20. A non-transitory computer-readable medium having stored thereon, computerexecutable instructions that when executed by an electronic device, causes the electronic device to execute operations, the operations comprising:monitoring, by use of a biometric sensor system, first sensor data associated with a first user; determining a world-view model of an environment corresponding to the first user; applying a multi-modal neural-language (MMNL) model on the determined world-view model and the monitored first sensor data; determining a situational context associated with the environment and the first user, based on the application of the MMNL model; receiving a first user-input based on the determined situational context; selecting, from a set of tasks, a first task based on the received first user-input, wherein the set of tasks is associated with an interaction of the first user and the environment; and controlling a first user-assistive actuator associated with the first user to execute at least partially the selected first task.

Images