Electronic device, method, and storage medium for customized care

Abstract

In the present disclosure, an electronic device may comprise: a memory including one or more storage media for storing instructions; and at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform at least one operation. The at least one operation may comprise an operation of acquiring biometric data by at least one sensor. The at least one operation may comprise an operation of acquiring language pattern data on the basis of natural language processing for speech of a call and / or an outgoing text message. The at least one operation may comprise an operation of acquiring image analysis data on the basis of analysis of a target image including at least a captured image. The at least one operation may comprise an operation of generating emotion or sentiment-related keywords on the basis of the acquired biometric data, the acquired language pattern data, or the image analysis data. The at least one operation may comprise an operation of performing inference on an emotional state or a sentimental state by analyzing the generated keywords in at least one on-device AI model. The at least one operation may comprise an operation of providing a linked service on the basis of a result according to the inference.

Description

Electronic devices, methods, and storage media for personalized care

[0001] The present disclosure relates to an electronic device, method, and storage medium for providing personalized mental health care in an on-device environment.

[0002] An artificial neural network (ANN) refers to a computational architecture that models the biological brain. Technologies such as deep learning and machine learning can be implemented based on ANNs. As an example of an ANN, a deep neural network may have a multilayer structure containing multiple layers.

[0003] AI models are being used in a variety of ways to analyze natural language, such as images or speech. For example, AI models can provide generative AI services such as interpretation, translation, summarization, image generation, or AR experiences.

[0004] AI models can be categorized as lightweight models (e.g., SLM) that operate on-device or large models (e.g., LLM) that operate in the cloud. In this case, AI services requiring relatively low computational throughput are preferably processed by lightweight models capable of fast processing, while AI services requiring high computational throughput are preferably processed by large models that can provide high quality but are slower. However, AI services provided in the cloud may carry the risk of personal information being exposed. Therefore, AI services based on data that can be classified as personal information must be provided by AI models based on the on-device environment.

[0005] The above information may be provided as background information to aid in understanding this document. None of the above is claimed to be prior art related to this document or can be used to determine prior art.

[0006] According to one embodiment, an electronic device may include a memory comprising one or more storage media storing instructions. The electronic device may include at least one processor comprising a processing circuit. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to perform at least one operation. The at least one operation may include obtaining biometric data by at least one sensor. The at least one operation may include obtaining language pattern data based on natural language processing of a call voice and / or an outgoing text message. The at least one operation may include obtaining image analysis data based on analysis of a target image including at least a photographed image. The at least one operation may include generating keywords related to emotions or sentiments based on the obtained biometric data, the obtained language pattern data, or the image analysis data. The at least one operation may include performing an inference on an emotional state or sentimental state by analyzing the generated keywords in at least one on-device AI model. The above at least one action may include an action of providing a linked service based on a result according to the above inference.

[0007] According to one embodiment, a storage medium may store computer-readable instructions. The instructions, when executed by at least a portion of at least one processor of the electronic device, may cause the electronic device to perform at least one operation. The at least one operation may include obtaining biometric data by at least one sensor. The at least one operation may include obtaining language pattern data based on natural language processing of a voice call and / or a text message. The at least one operation may include obtaining image analysis data based on analysis of a target image including at least a photographed image. The at least one operation may include generating keywords related to emotions or sentiments based on the obtained biometric data, the obtained language pattern data, or the image analysis data. The at least one operation may include analyzing the generated keywords in at least one on-device AI model to infer an emotional state or sentimental state. The at least one operation may include providing a linked service based on a result of the inference.

[0008] According to one embodiment, a method of operating an electronic device may include an operation of acquiring biometric data by at least one sensor. The method may include an operation of acquiring language pattern data based on natural language processing of a call voice and / or an outgoing text message. The method may include an operation of acquiring image analysis data based on an analysis of a target image including at least a photographed image. The method may include an operation of generating keywords related to emotions or sentiments based on the acquired biometric data, the acquired language pattern data, or the image analysis data. The method may include an operation of analyzing the generated keywords in at least one on-device AI model to infer an emotional state or sentimental state. The at least one operation may include an operation of providing a linked service based on a result of the inference.

[0009] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0010] FIG. 1 is a block diagram of an exemplary electronic device capable of performing the operations described herein.

[0011] FIG. 2 is an exemplary block diagram for providing a generative AI model in an electronic device according to one embodiment.

[0012] FIG. 3 is a block diagram of an exemplary AI system capable of performing the operations described in this document.

[0013] Figure 4 is a configuration diagram of a diagnostic system according to one embodiment.

[0014] FIG. 5 is a diagram for explaining an exemplary configuration and operation of a data processing module according to one embodiment.

[0015] FIG. 6 is a diagram illustrating a procedure for a user-customized care service in a diagnostic system according to one embodiment.

[0016] FIG. 7 is a control flowchart for providing a user-customized diagnostic service in an electronic device according to one embodiment.

[0017] FIG. 8A is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0018] FIG. 8b is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0019] FIG. 8c is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0020] FIG. 8d is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0021] FIG. 9 is a block diagram of an electronic device within a network environment according to various embodiments.

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0023] FIG. 1 is a block diagram of an exemplary electronic device (100) capable of performing the operations described in this document.

[0024] Referring to FIG. 1, the electronic device (100) may be one of various forms of electronic devices, such as a notebook (190), smartphones (191) having various form factors (e.g., a bar-type smartphone (191-1), a foldable-type smartphone (191-2), or a sliderable (or rollable) type smartphone (191-3)), a tablet (192), a cellular phone (not shown), and other similar computing devices (not shown). The components, their relationships, and their functions illustrated in FIG. 1 are exemplary only and do not limit the implementations described or claimed in this document. The electronic device (100) may be referred to as a mobile device, a user device, a multi-function device, a portable device, or a server.

[0025] The electronic device (100) may include components including at least one processor (110) (hereinafter, referred to as 'processor (110)'), at least one memory (120) (hereinafter, referred to as 'memory (120)'), at least one display (140) (hereinafter, referred to as 'display (140)'), at least one image sensor (150) (hereinafter, referred to as 'image sensor (150)'), at least one communication circuit (160) (hereinafter, referred to as 'communication circuit (160)'), and / or at least one sensor (170) (hereinafter, referred to as 'sensor (170)'). The components are merely exemplary. For example, the electronic device (100) may include other components (e.g., power management integrated circuitry (PMIC), audio processing circuitry, an antenna, a rechargeable battery, or an input / output interface). For example, some components may be omitted from the electronic device (100). For example, several components may be integrated into a single component. For example, the electronic device (100) may further include at least some of the configurations and / or functions not shown. At least some of the respective components of the electronic device shown (or not shown) may be operatively, functionally, and / or electrically connected to each other.

[0026] The processor (110) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing. The processor (110) may include at least one electrical circuit and may individually or collectively perform distributed processing of instructions (or programs, data, etc.) stored in the memory (120). The processor (110) may include a processor assembly including one or more processing circuits. The processor (110) may include any processing circuit operative to control the performance and operations of one or more components (e.g., the memory (120), the display (140), the image sensor (150), the communication circuit (160), and / or the sensor (170)) of the electronic device (100). For example, the processor (110) (e.g., an application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or chipset). For example, the processor (110) may be implemented with multiple cores (or at least one core circuit), multiple chips, or multiple chipsets. For example, the processor (110) may include one or more processing circuits. For example, the processor (110) may include one or more processing circuits configured to individually and / or collectively perform various functions of the present disclosure. As a non-limiting example, at least a portion of the processor (110) may be included in a first chip of the electronic device (100), and at least another portion of the processor (110) may be included in a second chip of the electronic device (100) that is different from the first chip of the electronic device (100).

[0027] For example, the processor (110) may include a central processing unit (CPU) (111), a graphics processing unit (GPU) (112), a neural processing unit (NPU) (113), an image signal processor (ISP) (114), a display controller (115), a memory controller (116), a storage controller (117), a communication processor (CP) (118), and / or a sensor interface (119). These components of the processor (110) are merely exemplary. For example, the processor (110) may further include other components. For example, some components of the processor (110) may be omitted from the processor (110). For example, some components of the processor (110) may be included as separate components of the electronic device (100) outside the processor (110). For example, some components of the processor (110) (e.g., memory controller (116)) may be included within other components (e.g., at least a portion of memory (120), an interface (e.g., available for connection to at least one component of the electronic device (100)), a display (140) and / or an image sensor (150)).

[0028] The processor (110) may cause other components of the electronic device (100) to perform various operations by executing instructions stored in the memory (120). The CPU (111) (or central processing circuit) may be configured to control components of the processor (110) based on the execution of instructions stored in the memory (120) (e.g., volatile memory (121) and / or non-volatile memory (122)). The GPU (112) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The NPU (113) (or neural processing circuit, or artificial intelligence (AI) chip) may be configured to execute operations for an AI model (e.g., convolution computation). The ISP (114) (or image signal processing circuit) may be configured to process a raw image acquired through the image sensor (150) into a format suitable for a component within the electronic device (100) or a component of the processor (110). The display controller (115) (or display control circuit, or DPU (display processing unit)) may be configured to process an image acquired from the CPU (111), GPU (112), ISP (114), or memory (120) (e.g., volatile memory (121)) into a format suitable for the display (140). The memory controller (116) (or memory control circuit) may be configured to control reading data from the volatile memory (121) and writing data to the volatile memory (121). The above storage controller (117) (or storage control circuit) may be configured to control reading data from the non-volatile memory (122) and writing data to the non-volatile memory (122).The CP (118) (communication processing circuit) may be configured to process data acquired from a component of the processor (110) into a format suitable for transmitting to another electronic device via the communication circuit (160), or to process data acquired from another electronic device via the communication circuit (160) into a format suitable for processing by a component of the processor (110). For example, the communication circuit (160) may include one or more communication circuits. The sensor interface (119) (or sensing data processing circuit, sensor hub) may be configured to process data on the state of the electronic device (100) and / or the state of the surroundings of the electronic device (100), acquired via the sensor (170), into a format suitable for a component of the processor (110).

[0029] The memory (120) may include one or more storage media (or one or more storage devices). For example, the memory (120) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory (122)) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory (121)) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (120) may include a cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (100). As a non-limiting example, the cache memory may be included within the processor (110). The memory (120) may be fixedly embedded within the electronic device (100) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that may be repeatedly inserted into and removed from the electronic device (100).

[0030] For example, the memory (120) may store one or more software applications, such as an operating system (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (110). For example, the memory (120) may store instructions callable by an application programming interface (API). For example, the memory (120) may store instructions within a library.

[0031] According to one example, the electronic device (100) can execute at least one instance of an AI model. The instance may be an object corresponding to a program (or application), such as an AI model, for example. The instance may be named a replica, a pod, a container, or a virtual machine, and there is no limitation on the name thereof. The number of instances may correspond to the size of a resource (e.g., a GPU (112) or an NPU (113)), and accordingly, the number of instances may be used interchangeably with the size of the resource, or the instances may be used interchangeably with the resource.

[0032] As an example, a plurality of user requests may be input to the electronic device (100). The user requests may be associated with a service. The user request may be processed by a first instance of a first AI model, and a first processing result may be provided from the first instance of the first AI model. The first processing result may be processed by a first instance of a second AI model, and accordingly, a second processing result may be provided by the first instance of the second AI model. By serial processing of the processing results, the first instance of the M-th AI model may receive and process the N-1-th processing result. The first instance of the M-th AI model may provide the N-th processing result as a response. Accordingly, a response corresponding to the user request may be provided.

[0033] Based on the above-described process, responses corresponding to each of a plurality of user requests may be provided. Meanwhile, since processing must be performed by an instance, the time for providing responses corresponding to each of a plurality of user requests (hereinafter referred to as “response time”) may take a relatively long time. The response time may affect the latency of the instance. In order to reduce the response time, the electronic device (100) may increase the number of instances of at least one AI model, which may be referred to as scaling out. However, there may be a limit to increasing the number of instances due to hardware and / or software constraints of the electronic device (100) and / or parameter restrictions of the AI ​​model (e.g., large language model (LLM)).

[0034] FIG. 2 is an exemplary block diagram for providing a generative artificial intelligence (AI) model in an electronic device (e.g., electronic device (100) of FIG. 1) (hereinafter referred to as 'electronic device (100)') according to one embodiment.

[0035] Referring to FIG. 2, the electronic device (100) may include a processor (110) (e.g., the processor (110) of FIG. 1), a memory (120) (e.g., the memory (120) of FIG. 1), and / or an interface (IF) (120). According to one example, all or part of the operations executed in the electronic device (100) may be executed in one or more external electronic devices. For example, when the electronic device (100) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (100) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to execute at least a part of the function or service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (100). The electronic device (100) may process the above results as is or additionally and provide them as at least part of a response to the request.

[0036] According to one example, an electronic device (100) can perform a service by linking with at least one AI system (200) (hereinafter referred to as 'AI system (200)'). The electronic device (100) may be implemented as, for example, a single entity, or may be implemented as a plurality of entities. For example, the electronic device (100) may operate an AI system (200) that can operate in an on-device environment, and there is no limitation on the form of implementation thereof. The AI ​​system (200) operated in the electronic device (100) in the on-device environment may include one or more AI models using machine learning and / or neural networks. For example, the AI ​​system (200) may execute an instance of at least one AI model. An instance can be an object corresponding to a program (or application), such as an AI model, for example, and can be named a replica, a pod, a container, or a virtual machine, and there is no limitation on the name. The number of instances can correspond to the size of a resource (e.g., a GPU), and accordingly, the number of instances can be used interchangeably with the size of the resource, or the instances can be used interchangeably with the resource. In the description to follow, it is assumed that, for example, one instance is executed for each of at least one AI models. As an instance corresponding to a specific AI model is executed, a resource of a predetermined size can be used. The resource of a predetermined size can mean, for example, a portion of a processor (110), a memory (120), and / or an IF (220).

[0037] The AI ​​system (200) may be based on natural language processing (NLP). NLP is a technology that allows an electronic device (200) to understand or process natural language input (hereinafter referred to as a "prompt") that can be expressed in the form of voice and / or text. The electronic device (200) can understand natural language through NLP and, based on this, determine human intention or convey information in a language that humans can understand. To understand human language, NLP can learn the order of words or tokens and predict the probability of the next word or token in a given text. A token is a basic unit for processing or understanding a prompt in an AI model. Key technologies of NLP include tokenization, part-of-speech tagging, syntax analysis, named entity recognition, and sentiment analysis for prompts corresponding to user input.

[0038] The I / F (220) can input a query (230) and transmit the input query (230) to the processor (110). The query (230) may correspond to, for example, one or more user requests. The query (230) may be a medium that guides the AI ​​system (200) to perform a task or generate a result in a desired direction. The query (230) may be the only window through which the user can communicate with the AI ​​system (200). The query (230) needs to be clear and specific in order to obtain an answer close to the desired result from the AI ​​system (200). According to one example, the I / F (220) may receive a response (240) (e.g., a summary message and / or a response message) processed by the AI ​​system (200) based on a query (230) (e.g., received messages), and output a response (240) converted into a natural language in a form that can be recognized by humans (e.g., text, image, audio, or video). The I / F (220) may input or output natural language in the form of voice and / or text through at least one component such as a keyboard, a touch panel, a display, and / or a speaker, for example.

[0039] The processor (110) may execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the electronic device (100) to which it is electrically connected. The processor (110) may perform various data processing or operations. As at least a part of the data processing or operations, the processor (110) may store instructions or data received from another component (e.g., an I / F (220)) in the memory (120) (e.g., a volatile memory, but without limitation). As at least a part of the data processing or operations, the processor (110) may process instructions or data stored in the memory (120) (e.g., a volatile memory, but without limitation). As at least a part of the data processing or operations, the processor (110) may store data resulting from processing instructions or data in the memory (120) (e.g., a non-volatile memory, but without limitation).

[0040] The memory (120) may store various data used by at least one component (e.g., processor (110) and / or I / F (220)) of the electronic device (100). The data may include, for example, software (e.g., program) and input data or output data for commands related thereto. The memory (120) may also store at least one AI model (e.g., sLM (small LM), LLM, LVM (large vision models), LMM (large multimodal models)) for executing one or more instances.

[0041] The memory (120) can store at least one instruction. The processor (110) can execute at least one instruction stored in the memory (120). When the at least one instruction is executed by the processor (110), the at least one instruction can cause the electronic device (100) to perform at least one operation. For example, as the at least one instruction is executed by the processor (110), at least one other component can be controlled, and / or various data processing or calculations can be performed. The performance of one operation by the processor (110) can mean, for example, that the operation is performed by (or under the control of) one entity included in the processor (110) (for example, the main processor, but without limitation). The performance of one operation can mean, for example, that a specific operation is performed by (or under the control of) multiple entities (for example, multiple processors). The fact that multiple operations are performed may mean, for example, that all of the multiple operations are performed by (or under the control of) one entity (e.g., but not limited to, a main processor (e.g., CPU (111) of FIG. 1).). The fact that multiple operations are performed may mean, for example, that some of the multiple operations are performed by at least one entity, and some of the remaining operations are performed by at least one other entity. At least one instruction causing the performance of one or more operations may be stored, for example, in one memory, or may be stored distributedly in each of a plurality of memories.

[0042] In the electronic device (100), the AI ​​system (200) may share resources (e.g., data processing or computational power) corresponding to part or all of at least one processor included in the processor (110) and / or resources (e.g., data recording area) corresponding to part or all of the memory (120). For example, the AI ​​system (200) may be operated by at least one of the CPU (111), the GPU (112), or the NPU (113). The AI ​​system (200) may be performed solely by the CPU (111), for example, by being assigned at least a portion of the memory (120). The AI ​​system (200) may be performed solely by the GPU (112), for example, by being assigned at least a portion of the memory (120). The AI ​​system (200) may be performed solely by the NPU (113), for example, by being assigned at least a portion of the memory (120). The AI ​​system (200) can be performed by a CPU (111) and a GPU (112) in cooperation, for example, by being assigned at least a portion of the memory (120). The AI ​​system (200) can be performed by a CPU (111) and an NPU (113) in cooperation, for example, by being assigned at least a portion of the memory (120). The AI ​​system (200) can be performed by a GPU (112) and an NPU (113) in cooperation, for example, by being assigned at least a portion of the memory (120). The AI ​​system (200) can be performed by a CPU (111), a GPU (112), and an NPU (113) in cooperation, for example, by being assigned at least a portion of the memory (120). The various embodiments to be described later in the present disclosure are not limited to the combination of components for performing the AI ​​system (200), and can be implemented and / or applied based on any combination.

[0043] FIG. 3 is a block diagram of an exemplary AI system (300) capable of performing the operations described in this document. The AI ​​system (300) may be a generative AI system, but will be referred to as the "AI system (300)" hereinafter.

[0044] Referring to FIG. 3, the AI ​​system (300) may include a User Query / Response Interface (310) (e.g., I / F (220) of FIG. 2) (hereinafter, referred to as 'I / F (310)'), an AI framework (320), a generative AI model (330), a database (340), or an Application / Service Component (350).

[0045] The I / F (310) can receive data or user input acquired or generated by an electronic device (e.g., the electronic device (100) of FIG. 1). The data acquired or generated by the electronic device (100) may include image or video data generated using a processor (e.g., the processor (110) of FIG. 1), values ​​transmitted through a sensor or sensor hub (e.g., external illuminance, an angle of the terminal, a display (e.g., the display (140) of FIG. 1) or the temperature of the electronic device (100), display (140) size or expansion / reduction information, an image captured by an image sensor (e.g., the image sensor (150) of FIG. 1), etc.). The user input may be in the form of natural language, touch coordinates or stylus coordinates acquired through a touch panel or digitizer included in the display (140), images, and / or videos, but is not limited thereto. In addition, context information may also be transmitted when transmitting the user input. Contextual information can include various additional information at the time of user input. For example, this additional information may include information about the application the user is currently using or the user's location. Furthermore, user input may also be a mixture of natural language, images, sounds, and contextual information described above. Furthermore, user input may also take the form of non-natural language, such as selecting a menu.

[0046] The I / F (310) can output the results of analyzing the output and / or input of the AI ​​system (300). The output may be in the form of natural language or specific content. The output may also be provided in the form of an action requested by the user. The output may also be provided in the form of a specific value specified by the user. The I / F (310) can output the results of the generative AI system (300) to the user. The output may be in the form of natural language or specific content. The output may also be provided in the form of an action requested by the user.

[0047] The AI ​​framework (320) can receive user input and coordinate and control each component necessary to carry out the user's intention based on the user's query. For example, the AI ​​framework (320) may include a prompt design component (321), an API / plug-in management component (323), or an output modification component (or refiner component) (325).

[0048] User input received from I / F (310) can be transmitted to a prompt design component (321). The prompt design component (321) can be used to generate a prompt suitable for inputting the user input into a generative AI model (330) (e.g., LLM, LVM, or LMM). The prompt design component (321) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (321) can access user preference data (343), a prompt library (341), and a knowledge component including prompt examples based on the user input to generate a prompt, and can transmit the generated prompt to the generative AI model (330), which is an LLM or LMM.

[0049] The API / plugin management component (323) may communicate with external information when a request for additional information is made when passing user input as input to the generative AI model (330). The API / plugin management component (323) establishes a channel for communicating with the outside of the AI ​​interface through the API, and enables access to various data sources (e.g., knowledge repositories (345)) through the established channel.

[0050] The API / plugin management component (323) may request the application / service component (323) to perform an action that ultimately requires user input, rather than an intermediate result, through an API when the action needs to be performed by the application or service. Information obtained from an external source may be used to generate a prompt in the prompt design component (321) along with user input, or may be passed as input to a generative AI model (330).

[0051] The output tuning component (325) (also referred to as a refiner component) can fine-tune or reprocess the output from the generative AI model (330). The output tuning component (325) can verify, for example, whether the content generated by the generative AI model (330) is irrelevant, biased, or harmful. The output tuning component (325) can determine to what extent the content matches the user's desired result and, if additional processing is required, can proceed with the process. The output tuning component (325) can additionally configure and provide the user with hints to avoid unwanted output.

[0052] A generative AI model (330) may generally refer to an AI neural network that creates new types of data based on user input information. A generative AI model (330) may include a model that generates images and / or a model that generates language. The model that generates images may include, for example, a generative adversarial network (GAN) or a variational autoencoder (VAE). The model that generates images may be, for example, a diffusion-based AI model that uses a VAE and a transformer structure. The model that generates language may be a model trained to output the most statistically appropriate output value based on input values. Representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there is also an LMM as an AI model (330) that can recognize various types of data input, such as text, images, voice, and video, and generate new data corresponding thereto.

[0053] Figure 4 is a configuration diagram of a diagnostic system (400) according to one embodiment.

[0054] Referring to FIG. 4, the diagnostic system (400) may be included in or executed by an electronic device (e.g., the electronic device (100) of FIG. 1). In one example, the diagnostic system (400) may correspond to an AI system (e.g., the AI ​​system (200) of FIG. 2) for providing personalized care related to mental health in an on-device environment by the electronic device (100). For example, the diagnostic system (400) may generate or pre-train one or more AI models for providing personalized care related to mental health in an on-device environment. The AI ​​system (200) may include, for example, one or more AI models, each of which may perform its own instantiation. The diagnostic system (400) may include a data collection module (410), a data processing module (420), a data storage module (430), and / or a data processing module (440). According to one example, at least one AI model may be applied to each of the data collection module (410), data processing module (420), data storage module (430), and / or data processing module (440) included in the diagnostic system (400).

[0055] The data collection module (410) can collect various types of emotional data related to the user's mental health. The emotional data may include data that can predict and / or infer emotional states, such as emotion, sentiment, and / or mood, which are important factors that can affect human mental health. Emotion may refer to a mental and / or physiological emotional state, such as temporary anger, fear, joy, sadness, and / or surprise, caused by a relatively strong, short-term stimulus. Emotion may, for example, be related to mood, temperament, and / or personality as a subjective experience. Affect or mood may refer to a mental and / or physiological emotional state, such as exhilaration or depression, caused by a relatively weak, persistent stimulus, although not direct.

[0056] According to one example, the data collection module (410) may collect biometric data, behavioral data, and / or creative work-related data as emotional data. The biometric data may be data regarding various forms of physiological responses that may appear in a user, reflecting emotional states such as emotions, moods, and / or feelings. The behavioral data may be data regarding various forms of actions (e.g., speech, writing, and / or actions) that may appear in a user, reflecting emotional states such as emotions, moods, and / or feelings. The creative work-related data may be data regarding records (e.g., photos, videos, or diaries) created by a user, reflecting emotional states such as emotions, moods, and / or feelings. The data collection module (410) may store emotional data including biometric data, behavioral data, and / or creative work-related data in the data storage module (430).

[0057] For example, the data collection module (410) can acquire biometric data. The biometric data may be data regarding various types of physiological responses exhibited by the user according to emotional states such as emotions, moods, and / or feelings. The biometric data may be one of the life activities occurring inside the body, and may be electrical signals in the form of current or voltage generated by nerve cells or muscle cells. The biometric data may be responses occurring in the body, such as electroencephalogram (EEG), electrocardiogram (ECG), skin conductance (SKD), electroocular electroencephalogram (EOE), electromyogram (EMG), magnetoencephalogram (MEG), pulse wave, blood pressure, heartbeat sound, stress index, or oxygen saturation. The data collection module (410) can acquire biometric data from various types of sensors. At least one sensor for acquiring biometric data may be provided in an external device (e.g., a wearable / hearable device), for example. At least one sensor for acquiring biometric data may be provided in the electronic device (100), for example, the sensor (170) of FIG. 1. At least one sensor for acquiring biometric data may be provided partially in an external device (e.g., a wearable / hearable device) and partially in an electronic device (100) (e.g., a sensor (170) of FIG. 1). A wearable device or hearable device may be worn by a user on the wrist, finger, or head to measure data related to biometric responses or movements. For example, a wearable device may include a smart watch or a smart ring. A hearable device may include a smart headset or a virtual reality (VR) headset. Behavioral data may include, for example, sleep pattern data or exercise intensity data collected by a wearable / hearable device during daily life.

[0058] For example, the data collection module (410) may obtain behavioral data. The behavioral data may be data that reflects various forms of behavioral responses from a user depending on emotional states such as emotions, moods, and / or feelings. The data collection module (410) may obtain data related to the use of natural language, such as language and / or text, which may externally expose a human's emotional state (hereinafter referred to as "natural language data") as behavioral data. The data collection module (410) may collect, as natural language data, data regarding voice that may change due to emotional states, such as intonation, word usage, trembling, or speech speed when speaking for a call or conversation. The data collection module (410) may collect, as natural language data, data regarding text that may change due to emotional states, such as sentence structures or word usage entered by a user in an application that provides services such as chatting, text transmission, or memos (hereinafter referred to as "text data"). The data collection module (410) can obtain voice data from the user's voice input through a microphone provided in the electronic device (100). The data collection module (410) can obtain character data from characters input through an input means (e.g., a touch panel, a keypad) provided in the electronic device (100).

[0059] According to one example, the data collection module (410) can obtain behavioral data. The behavioral data may be data that can reflect various types of behavioral responses exhibited by the user according to emotional states such as emotions, moods, and / or feelings. The data collection module (410) can obtain data related to the user's movements or actions that can externally expose a human emotional state (hereinafter referred to as "motion data") as behavioral data. The data collection module (410) may include data regarding actions that can change due to emotional states, such as gait, walking speed, running speed, sleep, or exercise intensity. The data collection module (410) can obtain motion data from various types of sensors. At least one sensor for obtaining motion data may be provided, for example, in an external device (e.g., a wearable / hearable device). At least one sensor for obtaining motion data may be provided, for example, inside the electronic device (100) (e.g., the sensor (170) of FIG. 1). At least one sensor for acquiring motion data may be provided in part on an external device (e.g., a wearable / hearable device) and in part on the electronic device (100) (e.g., the sensor (170) of FIG. 1). The wearable device or hearable device may be worn by a user on the wrist, finger, or head to measure motion data. For example, the wearable device may include a smartwatch or a smart ring. The hearable device may include a smart headset or a VR headset. The motion data may include, for example, sleep pattern data or exercise intensity data collected by the wearable / hearable device during daily life.

[0060] In one example, the data collection module (410) may acquire creative data. The creative data may be data related to records (e.g., photos, videos, or diaries) that may reflect the user's emotional state, such as emotions, sentiments, and / or moods. The data collection module (410) may acquire creative data from an application (e.g., a gallery application) running on the electronic device (100) that allows creation of creative works, such as photos, videos, or writings.

[0061] The data processing module (420) can analyze and / or process the emotional data collected by the data collection module (410) to obtain usable data. For example, the data processing module (420) can include at least one application that can process the emotional data collected by the data collection module (410) to derive features. For example, the data processing module (420) can include a wearable application. The wearable application can periodically request biometric data from one or more wearable devices connected based on a predetermined communication method (e.g., BT, WiFi, UWB). In response to the request, the wearable application can process the biometric data received from one or more wearable devices to obtain emotional data corresponding to predetermined features that can affect the user's emotional state, such as emotions, moods, and / or feelings. For example, the data processing module (420) can include a gallery application. The gallery application can analyze photos and / or video data taken by the user during their daily lives to obtain emotional data corresponding to characteristics that may influence the user's emotional state, such as emotions, sentiments, and / or mood. The emotional data may include biometric data, behavioral data, or creative data. The available data may include data that can be used to predict and / or infer the user's emotional state, such as emotions, sentiments, and / or mood. The available data may also include data that can be used to predict and / or infer the user's mental health status.

[0062] The data processing model (420) may be equipped with a digital signal processor (DSP). The DSP may, for example, be involved in driving a data collection module (410) that collects conversation detection and / or language pattern data occurring in daily life. The DSP may provide an operation to extract or generate (hereinafter referred to as “generate”) keywords that may influence a user’s emotional state, such as sentiment, mood, and / or mood, at low power. The DSP may include a keyword spotter (KWS). The KWS may, for example, analyze emotional data to detect keywords that may influence a user’s emotional state, mental health state, and / or physical health state.

[0063] The data processing model (420) may include a Keyword Spotter (KWS) for analyzing emotional data to detect keywords that may influence the user's emotional state, mental health, and / or physical health. The KWS may be an AI model that has previously learned words and language patterns that may influence the user's emotional state, mental health, and / or physical health. The data processing module (420) may analyze and / or process the emotional data and store the acquired available data in a data storage module (430) in a predetermined batch size.

[0064] In one example, the data processing module (420) may obtain available data to be utilized for predicting and / or inferring the user's emotional state or mental health state based on the biometric data collected by the data collection module (410). The data processing module (420) may obtain available data to be utilized for predicting and / or inferring the user's physical health state (e.g., panic attack, cardiac arrest) based on the biometric data collected by the data collection module (410). For example, in order to analyze the biometric data, the data processing module (420) may apply an AI model trained based on deep learning and / or machine learning. In one example, the data processing module (420) may analyze the biometric data to extract keywords for inferring the emotional state, mental health state, and / or physical health state as available data. In one example, the data processing module (420) may analyze the biometric data to extract patterns of changes in emotional states, such as emotions, moods, and / or sentiments, as available data.

[0065] In one example, the data processing module (420) may obtain available data to be utilized for predicting and / or inferring the user's emotional state or mental health state based on the behavioral data collected by the data collection module (410). The behavioral data may include, for example, language pattern data related to language characteristics, such as words, intonation, or tremors used during voice calls or conversations. The behavioral data may include, for example, text pattern data related to text input by the user through an application that provides services such as chatting, texting, or memo. The data processing module (420) may obtain available data to be utilized for predicting and / or inferring the user's physical health state (e.g., panic attack, cardiac arrest) based on the behavioral data collected by the data collection module (410). For example, to analyze the behavioral data, the data processing module (420) may apply an AI model trained based on deep learning and / or machine learning. For example, the data processing module (420) may analyze behavioral data to extract keywords for inferring emotional states, mental health states, and / or physical health states as available data. For example, the data processing module (420) may analyze behavioral data to extract patterns of change in emotional states, such as emotions, moods, and / or feelings, as available data.

[0066] According to one example, the data processing module (420) can analyze the natural language data collected by the data collection module (410) to obtain available data (e.g., keywords, natural language pattern data) related to the user's emotional state. The data processing module (420) can analyze the natural language data based on natural language processing (NLP) technology learned by deep learning and / or machine learning. The data processing module (420) can, for example, analyze language data included in the natural language data to obtain available data (e.g., keywords, language pattern data) that can describe the user's emotional state. The data processing module (420) can, for example, analyze text data included in the natural language data to obtain available data (e.g., keywords, text pattern data) that can describe the user's emotional state.

[0067] According to one example, the data processing module (420) can analyze the motion data collected by the data collection module (410) to obtain available data (e.g., keywords, behavior pattern data) related to the user's emotional state, mental health state, and / or physical health state. The data processing module (420) can analyze the motion data based on an AI model learned by deep learning and / or machine learning for motion analysis. The data processing module (420) can, for example, analyze the movement pattern (e.g., gait, walking speed, running speed, sleep, or exercise intensity) based on the motion data to obtain available data (e.g., keywords, behavior pattern data) that can describe the user's emotional state, mental health state, and / or physical health state.

[0068] In one example, the data processing module (420) may analyze creation-related data (e.g., photos, videos, diaries) collected by the data collection module (410) to obtain available data (e.g., keywords) related to the user's emotional state and / or mental health state. The data processing module (420) may analyze creation-related data based on image analysis techniques learned through deep learning and / or machine learning. The data processing module (420) may, for example, analyze creation-related data to obtain available data (e.g., keywords) that may describe the user's emotional state and / or mental health state.

[0069] The data storage module (430) can store biometric data, behavioral data, and / or creation-related data collected by the data collection module (410). The data storage module (430) can transmit biometric data, behavioral data, and / or creation-related data collected by the data collection module (410) at the request of the data processing model (420). The data storage model (430) can store available data, such as keywords, behavioral pattern data, and natural language pattern data (e.g., language pattern data, character pattern data), obtained by the data processing model (420). The data storage module (430) can transmit available data obtained by the data processing model (420) at the request of the data processing module (440).

[0070] The data processing module (440) can check the amount of available data stored in the data storage module (430) at specific intervals (e.g., daily, weekly, or monthly). The data processing module (440) can, for example, perform inferences regarding emotional states, mental health states, and / or physical health states based on available data (e.g., keywords, behavioral pattern data, natural language pattern data (e.g., language pattern data, character pattern data)) stored in the data storage module (430) above a critical level.

[0071] For example, the data processing module (440) can calculate scores for emotional state, mental health state, and / or physical health state according to the type of available data. For example, the data processing module (440) can calculate a first score for emotional state and / or mental health state based on keywords and / or language pattern data according to the user's speech. For example, the data processing module (440) can calculate a second score for emotional state and / or mental health state based on keywords and / or language pattern data according to the user's character usage. For example, the data processing module (440) can calculate a third score for emotional state, mental health state, and / or physical health state based on keywords and / or behavioral pattern data according to the user's biometric data. For example, the data processing module (440) can calculate a fourth score for emotional state and / or mental health state based on keywords according to the user's creative data (e.g., photos, videos, diaries).

[0072] In addition to the first to fourth scores, the data processing module (440) may additionally consider the user profile (e.g., gender, age, family relationships, personality type, religion, nationality, etc.) to determine a final score that infers the user's emotional state, mental health, and / or physical health. The data processing module (440) may derive a diagnosis result for the user based on the final score. The data processing module (440) may provide a customized service to the user based on the derived diagnosis result. For example, the data processing module (440) may recommend exercise or a conversation with a close friend. For example, the data processing module (440) may provide a report on the user's daily / weekly / monthly emotional state. For example, the data processing module (440) may provide a method of coping with panic disorder based on the user's physical health. For example, the data processing module (440) may provide an assessment of work stress and a method of relieving stress. For example, the data processing module (440) may recommend an appointment with a doctor or provide a community connection service for people with similar emotional states.

[0073] FIG. 5 is a drawing for explaining an exemplary configuration and operation of a data processing module (e.g., the data processing module (440) of FIG. 4) according to one embodiment.

[0074] Referring to FIG. 5, the data processing module (440) may include a service framework (510) and / or a personal spiritual health (PSH) neural network core (520). The service framework (510) may correspond to, for example, the data collection module (410) and / or the data processing module (420) described in FIG. 4. The PSH neural network core (520) may correspond to, for example, the data processing module (440) described in FIG. 4.

[0075] The service framework (510) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0076] According to one example, the service framework (510) can collect various types of emotional data related to the user's mental health (e.g., biometric data, behavioral data, creative data). The service framework (510) can receive biometric data and / or behavioral data from, for example, a wearable and / or hearable device (530). Although not shown, the service framework (510) can receive biometric data and / or behavioral data from, for example, at least one sensor provided internally. The biometric data can be data regarding bodily reactions, such as electroencephalogram (EEG), electrocardiogram (ECG), skin conductance (SKD), electroocular electrocardiogram (EOG), electromyogram (EMG), magnetoencephalogram (MEG), pulse wave, blood pressure, heartbeat sound, stress index, or oxygen saturation. The behavioral data can include, for example, sleep pattern data or exercise intensity data that can be collected during daily life. The service framework (510) can receive creative data from, for example, a specific application (540), such as a gallery. Creative-related data may be, for example, data about records (e.g., photographs, videos, or diaries) that may reflect the user's emotional state, such as emotions, sentiments, and / or moods.

[0077] The service framework (510) may include a user interactor (511), a data scheduling manager (513), or a data access manager (515) for a spiritual health service. According to an example, the mental health service may receive emotional data such as biometric data, voice data, text data, or image data from an application (540) in batches, store them in a memory (550), and periodically call an AI model to receive an inference result (570) based on the emotional data to provide a report related to mental health.

[0078] According to an example, the user interactor (511) may provide the user with an appropriate pop-up based on the score when there is an issue with the user's emotional state and / or mental health, such as emotions, moods, and / or feelings. When there is an issue with the user's emotional state and / or mental health, the user interactor (511) may inform the user of the issue and provide a solution. The user interactor (511) may be, for example, an application, an alarm, or an AI user interface (UI) that interacts with the user (see FIGS. 8A, 8B, 8C, and 8D). The user interactor (511) may provide different solutions (e.g., recommending a doctor visit, providing a meditation video, suggesting the cause of an argument with a friend and what to say, etc.) depending on the severity of the user's emotional state and / or mental health.

[0079] According to one embodiment, the data scheduling manager (513) can manage the cycle at which the corresponding module is called to store collected emotional data, such as biometric data, language data, text data, or image data (or content data). For example, if the data scheduling manager (513) provides a specific function (e.g., a real-time alarm for panic disorder) (see FIG. 8c), the cycle can be very short. In other words, a system capable of responding in real time can be configured. For example, when providing a service that reports emotional states at a specific cycle (e.g., daily, weekly, or monthly), the data scheduling manager (513) can configure a system capable of evaluating emotional states at a specific cycle (see FIG. 8a).

[0080] According to one embodiment, the data access manager (515) can access a memory (550) in which emotional data collected for each period, which can be managed for each service by the data scheduling manager (513), is stored, and acquire emotional data to be used for the service. The data access manager (515) can transmit the acquired emotional data to the PSH neural network core (520).

[0081] The PSH neural network core (520) may receive collected emotional data, such as biometric data, language data, text data, or image data (or content data), from a module providing mental health services provided in the service framework (510). The PSH neural network core (520) may include an AI model capable of performing inference based on the emotional data received from the service framework (510) and assigning a score to the user's emotional state, mental health state, and / or physical health state.

[0082] For example, the PSH neural network core (520) may include a language model (521), an image model (523), a sensor model (525), or a final classifier model (527). Depending on the type of emotional data transmitted from the service framework (510), the PSH neural network core (520) may use one or all of the language model (521), the image model (523), or the sensor model (525), or may use a hybrid model combining multiple models.

[0083] The language model (521) may be an AI model that performs inference to assign a score corresponding to the user's emotional state, mental health state, and / or physical health state based on the user's language pattern data received from the service framework (510), for example.

[0084] The image model (523) may be an AI model that performs inference to assign a score corresponding to the user's emotional state, mental health state, and / or physical health state based on creative work-related data such as photos, videos, or diaries received from the service framework (510), for example.

[0085] The sensor model (525) may be an AI model that performs inference to assign a score corresponding to the user's emotional state, mental health state, and / or physical health state based on biometric data received from the service framework (510), for example.

[0086] The classification model (527) may be provided with a score inferred in response to the user's emotional state, mental health state, and / or physical health state, for example, by a language model (521), an image model (523), or a sensor model (525). The classification model (527) may be provided with, for example, a user profile (e.g., gender, age, family relationship, personality type, religion, nationality, etc.) and / or user data (560). The user profile (560) may be used as auxiliary data for analyzing the user's emotional state, mental health state, and / or physical health state. The classification model (527) may infer a score corresponding to the final result (570) by performing a softmax on the final result regarding the user's emotional state, mental health state, and / or physical health state, for example. The classification model (527) may classify the user's state by synthesizing the user's emotional state, mental health state, and / or physical health state, for example.

[0087] FIG. 6 is a drawing for explaining a procedure for a user-customized care service in a diagnostic system (e.g., the diagnostic system (400) of FIG. 4) according to one embodiment.

[0088] Referring to FIG. 6, the electronic device (100) can provide a personalized care service. The personalized care service provided by the electronic device (100) can include, for example, a collection step (610), a keyword extraction step (620), an analysis step (630), and a diagnosis step (640).

[0089] According to one example, the collection step (610) is an operational step of collecting various types of emotional data from wearable devices such as smartwatches or smart rings, hearable devices such as smart headsets and VR headsets, or applications (611). The emotional data may include data that can predict and / or infer emotional states such as emotions, sentiments, and / or moods, which are important factors that can affect human mental health. The emotional data may reflect the user's emotional states such as emotions, sentiments, and / or moods. According to one example, the emotional data may include biometric data (613), habit data (e.g., language, characters) (615), or content data (e.g., data related to creative works such as photos, videos, and diaries) (617). Biometric data (613) may be, for example, electrical signals generated by the body, such as electroencephalogram, electrocardiogram, skin conductance, electroocular conductance, electromyogram, magnetoencephalogram, magnetocardiogram, pulse wave, blood pressure, heartbeat sound, stress index, or oxygen saturation. Habit data (615) may be, for example, linguistic data including characteristics such as intonation, word usage, tremors, or speech speed that may change due to a call or emotional state. Habit data (615) may be, for example, text data including characteristics such as sentence structure or word usage that may change due to an emotional state. Habit data (615) may be, for example, behavioral data including characteristics related to gait, walking speed, running speed, sleep, or exercise intensity that may change due to an emotional state. Content data (617) may be, for example, creative data regarding records such as photos, videos, or diaries that may reflect an emotional state such as a user's emotions, sentiments, and / or moods.

[0090] In one example, the keyword extraction step (620) is an operational step for analyzing and / or processing the collected emotional data to extract available data including keywords and / or pattern data (e.g., voice pattern data, text pattern data, behavioral pattern data). The available data may describe a user's state, such as emotion (623), sentiment (625), and / or mental state (627). The available data may be utilized to infer the user's emotional state and / or mental health state. For example, the keyword extraction step (620) may be performed by an AI keyword extraction model (621).

[0091] According to one example, the analysis step (630) includes an operation step of analyzing the user's emotional state (633) by performing inference based on available data including keywords and / or pattern data (e.g., voice pattern data, text pattern data, behavioral pattern data) generated in the keyword extraction step (620). For example, the operation step of analyzing the emotional state (633) may be performed by an AI analysis model (631).

[0092] In one example, the analysis step (630) includes an operation step of analyzing the user's emotional state (635) by performing inference based on available data including keywords and / or pattern data (e.g., voice pattern data, text pattern data, behavioral pattern data) generated in the keyword extraction step (620). For example, the operation step of analyzing the emotional state (635) may be performed by an AI analysis model (631).

[0093] In one example, the analysis step (630) includes an action step of analyzing the user's mental state (637) by performing inference based on available data including keywords and / or pattern data (e.g., voice pattern data, text pattern data, behavioral pattern data) generated in the keyword extraction step (620). For example, the action step of analyzing the mental state (637) may be performed by an AI analysis model (631).

[0094] For example, the diagnosis step (640) may be an action step that determines a customized service based on the inference results and the user profile, and performs a diagnosis (643), treatment (645), or feedback (647) regarding emotional state, mental health, and / or physical health in response to the determined customized service. The user profile may include information such as gender, age, family relationships, personality type, religion, or nationality. For example, a customized service may recommend exercise to the user or a conversation with a close friend. For example, a customized service may provide the user with a report on daily / weekly / monthly emotional state. For example, a customized service may provide a coping method for panic disorder by considering the user's physical health. For example, a customized service may provide an assessment of work stress and a stress relief method. For example, a customized service may recommend a consultation with a doctor or provide a community connection service for people with similar emotional states.

[0095] FIG. 7 is a control flowchart for providing a user-customized diagnostic service in an electronic device (e.g., the electronic device (100) of FIG. 1) according to one embodiment.

[0096] The user-tailored diagnostic service proposed in FIG. 7 may be provided by a diagnostic system (e.g., the diagnostic system (400) of FIG. 4) that may be included in or executed by the electronic device (100). The diagnostic system (400) may correspond to an AI system (e.g., the AI ​​system (200) of FIG. 2) for providing personalized care related to mental health in an on-device environment by the electronic device (100). For example, the diagnostic system (400) may generate or pre-train one or more AI models for providing personalized care related to mental health in an on-device environment. The AI ​​system (200) may include, for example, one or more AI models, each of which may perform unique instantiation.

[0097] Referring to FIG. 7, the electronic device (100) may, in operation 711, collect various types of emotional data related to the user's mental health. The emotional data may include data that can predict and / or infer emotional states, such as emotions, moods, and / or feelings, which are important factors that can affect human mental health. The emotional data may reflect the user's emotional states, such as emotions, moods, and / or feelings. In one example, the emotional data may include biometric data, behavioral data, and / or creative work-related data.

[0098] For example, the electronic device (100) may acquire biometric data through an external device such as a smartwatch, a smart ring, a smart headset, or a VR headset, or at least one sensor provided internally. The biometric data may be, for example, an electrical signal generated by the body, such as an electroencephalogram (EEG), an electrocardiogram (ECG), a skin conductance (SKD), an electrooculogram (EOG), an electromyogram (EMG), a magnetoencephalogram (MEG), a magnetocardiogram (MCG), a pulse wave, blood pressure, a heartbeat, a stress index, or oxygen saturation.

[0099] For example, the electronic device (100) may acquire language data through a microphone, text data through an input means such as a touch panel or a keypad, and behavioral data through at least one sensor. The language data may include characteristics such as intonation, word usage, trembling, or speech rate, which may change due to a call or emotional state. The text data may include characteristics such as sentence structure or word usage, which may change due to an emotional state. The behavioral data may include characteristics related to gait, walking speed, running speed, sleep, or exercise intensity, which may change due to an emotional state, for example.

[0100] In one example, the electronic device (100) may obtain creative data from a corresponding application (e.g., a gallery application). The creative data may include, for example, data regarding photographs, videos, or records such as diaries that may reflect the user's emotional state, such as emotions, sentiments, and / or mood.

[0101] The electronic device (100) may, in operation 713, analyze and / or process the collected emotional data to extract and store available data including keywords and / or pattern data (e.g., voice pattern data, text pattern data, behavioral pattern data). The available data may be utilized to predict and / or infer the user's emotional state, such as sentiment, mood, and / or mood. The available data may be utilized to predict and / or infer the user's mental health state. The electronic device (100) may include, for example, a KWS for analyzing the emotional data to detect keywords that may affect the user's emotional state, mental health state, and / or physical health state. The KWS may be an AI model that has learned words and language patterns that may affect the user's emotional state, mental health state, and / or physical health state in advance. The electronic device (100) may store the available data obtained by analyzing and / or processing the emotional data in a batch size.

[0102] According to one example, the electronic device (100) may obtain available data including keywords and / or patterns of change in emotional states for predicting and / or inferring the emotional state or mental health state of the user based on the collected biometric data. The electronic device (100) may obtain available data including keywords and / or patterns of change in emotional states for predicting and / or inferring the physical health state (e.g., panic attack, cardiac arrest) of the user based on the collected biometric data.

[0103] According to one example, the electronic device (100) may obtain available data including keywords and / or patterns of change in emotional states for predicting and / or inferring the user's emotional state or mental health state based on collected behavioral data. The electronic device (100) may obtain available data including keywords and / or patterns of change in emotional states for predicting and / or inferring the user's physical health state (e.g., panic attack, cardiac arrest) based on collected behavioral data.

[0104] For example, the electronic device (100) may analyze the collected language data to obtain keywords or language pattern data that can describe the user's emotional state as available data. The electronic device (100) may analyze the collected text data to obtain keywords or text pattern data that can describe the user's emotional state as available data.

[0105] According to one example, the electronic device (100) can analyze the movement pattern (e.g., gait, walking speed, running speed, sleep, or exercise intensity) based on the collected motion data to obtain keywords and behavior pattern data that can describe the user's emotional state, mental health state, and / or physical health state as available data.

[0106] According to one example, the electronic device (100) can analyze creative data such as collected photos, videos, or diaries to obtain keywords that can describe the user's emotional state and / or mental health state as available data.

[0107] The electronic device (100) may determine, at operation 715, whether an inference event occurs. The inference event may occur, for example, at specific intervals (e.g., daily, weekly, monthly). The inference event may occur, for example, when a threshold level of available data (e.g., keywords, behavioral pattern data, language pattern data, character pattern data) is stored. The inference event may occur, for example, in an emergency situation, such as a physical abnormality.

[0108] When an inference event occurs, the electronic device (100) may, in operation 717, obtain available data corresponding to event-related feature data for performing inference regarding an emotional state, mental health state, and / or physical health state.

[0109] The electronic device (100) may perform event-related inference at operation 719. For example, the electronic device (100) may perform inference regarding emotional state, mental health state, and / or physical health state by inputting available data extracted from biometric data and keywords and language pattern data generated from voice data.

[0110] The electronic device (100) may obtain a mental health-related score based on the inference result and the user profile in operation 721. In one example, the electronic device (100) may calculate a score regarding the emotional state, mental health state, and / or physical health state depending on the type of available data. For example, the electronic device (100) may calculate a first score regarding the emotional state and / or mental health state based on keywords and / or language pattern data according to the user's utterance. For example, the electronic device (100) may calculate a second score regarding the emotional state and / or mental health state based on keywords and / or language pattern data according to the user's character usage. For example, the electronic device (100) may calculate a third score regarding the emotional state, mental health state, and / or physical health state based on keywords and / or behavioral pattern data according to the user's biometric data. For example, the electronic device (100) can calculate a fourth score regarding the emotional state and / or mental health state based on keywords according to the user's creative related data (e.g., photos, videos, diary).

[0111] According to one example, the electronic device (100) may determine a final score that infers the user's emotional state, mental health state, and / or physical health state by additionally considering the user profile (e.g., gender, age, family relationship, personality type, religion, nationality, etc.) in addition to the first to fourth scores.

[0112] The electronic device (100) can derive a diagnosis result for the user based on the final score in operation 723. The electronic device (100) can determine a customized service for the user based on the derived diagnosis result.

[0113] The electronic device (100) may, at operation 725, provide a report based on a personalized service. For example, the electronic device (100) may recommend exercise to the user or a conversation with a close friend. For example, the electronic device (100) may provide the user with a report on daily / weekly / monthly emotional states. For example, the electronic device (00) may provide a method of coping with panic disorder by considering the user's physical health. For example, the electronic device (100) may provide an assessment of work-related stress and stress relief methods. For example, the electronic device (100) may recommend a consultation with a doctor or provide a community connection service for people with similar emotional states.

[0114] FIG. 8A is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0115] Referring to FIG. 8A, an AI model (810) included in an electronic device (e.g., electronic device (100) of FIG. 1) providing an AI service in an on-device environment can analyze emotional data periodically and / or aperiodically. The emotional data can include, for example, biometric data, natural language data (e.g., voice recording, text recording), and / or content data. The AI ​​model (810) can analyze the emotional data to obtain available data. The available data can be data related to an emotional state, such as a user's emotion, sentiment, and / or mood, which can be obtained by analyzing the emotional data. The available data can include, for example, keywords, language pattern data, text pattern data, or behavioral pattern data that describe an emotional state, such as a user's emotion, sentiment, and / or mood. The AI ​​model (810) can infer an emotional state, such as a user's emotion, sentiment, and / or mood, based on the available data. For example, the AI ​​model (810) may provide the user with a customized suggestion corresponding to the user's emotional state, which is the result of the inference (811). For example, the AI ​​model (810) may provide the user with data regarding the user's emotional state, which is the result of the inference, at a specific interval (e.g., daily, weekly, or monthly) (813).

[0116] below is an example of how an AI model (810) provides customized feedback and advice on improving lifestyle habits based on a user's emotional state diary and analysis results.

[0117] By recording an emotional state log, the user's current emotional state and past data are integrated to analyze the trend of changes in the user's emotional state. The user's daily, weekly, and monthly emotional state changes are visually provided in the form of graphs and charts through electronic devices. Personalized advice is provided. Additional support such as meditation apps and psychological counseling services are suggested as needed. Personalized health advice such as stress management, sleep improvement, and exercise are provided based on the analyzed data. User feedback: By adding feedback to the user through natural language such as voice and text, analyzing it, and adding it to the emotional state log, a customized service is provided. Continuous status monitoring: The user's emotional state is continuously monitored to enable immediate support when necessary. User community connection: Provides a community where users who experience similar emotional states can connect and support each other. Continuous improvement: The emotional state log and feedback system are continuously improved based on user feedback and new data.

[0118] According to the above , the AI ​​model (810) can provide a linked service by analyzing biometric data, voice data, text data, or content data (e.g., taken photos, videos, diaries) acquired during the user's daily life to infer the user's emotional state.

[0119] FIG. 8b is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0120] Referring to FIG. 8B, an AI model (820) included in an electronic device (e.g., electronic device (100) of FIG. 1) providing an AI service in an on-device environment can collect voice data according to a voice conversation during a user's social interaction (e.g., a conversation with a friend). The AI ​​model (820) can analyze the collected voice data to obtain voice pattern data such as intonation, word usage, tremors, or speech speed. The AI ​​model (820) can infer the user's emotional state based on the voice pattern data. For example, the AI ​​model (820) can recommend an appropriate response (e.g., a method of expressing agreement, a language of empathy) to the user in natural language during a conversation based on the user's emotional state. This can enable smooth interaction with others by taking the user's emotional state into consideration. This can provide a reporting function to easily check the user's social interaction history.

[0121] For example, the AI ​​model (820) can provide customized services to users based on their inferred emotional states. For example, the AI ​​model (820) can analyze the user's emotional state in various situations and provide appropriate feedback. This can help maintain a healthy emotional state.

[0122] In the illustrated example, a user argues with a friend yesterday. The AI ​​model (820) can collect voice data corresponding to the conversation the user had with the friend. The AI ​​model (820) can analyze the collected voice data to obtain the user's voice pattern data, such as intonation, word usage, trembling, or speech rate. Based on the user's voice pattern data, the AI ​​model (820) can infer the user's emotional state due to the argument with the friend. The AI ​​model (820) can not only diagnose the user's emotional state by considering the inferred emotional state, but can also provide an appropriate solution corresponding to the diagnosis. For example, the AI ​​model (820) can objectively analyze the cause of the argument with a friend and provide an appropriate solution for improving the relationship. For example, the AI ​​model (820) can analyze / evaluate communication styles in daily life and provide customized advice based on the analysis. For example, the AI ​​model (820) can say, "I was sad because I had a fight with my friend yesterday, but I think my friend was acting like ~ in his mind!" You can provide users with a comment using voice or text, such as, 'If you tell your friend ~, you might be able to open up to each other!'

[0123] FIG. 8c is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0124] Referring to FIG. 8c, an AI model (835) included in an electronic device (e.g., electronic device (100) of FIG. 1) that provides AI services in an on-device environment can infer that the user is in an emergency situation by analyzing biometric data and / or voice data. If the occurrence of an emergency situation is predicted through inference, the AI ​​model (835) can provide a linked service. For example, the AI ​​model (835) can analyze the user's biometric data to recognize that there is an abnormality in the user's physical health, and can assume a situation in which voice data corresponding to the user's words (e.g., "I feel anxious, my chest feels tight!") (833) is obtained (831). In this case, the AI ​​model (835) can infer the user's physical health state based on the collected biometric data and voice data. The AI ​​model (835) can utilize personal data, such as the user's medical history, to infer the user's physical health state. For example, if the AI ​​model infers that there is a physical abnormality due to panic disorder based on the collected biometric data and voice data, the AI ​​model can provide a linked service.

[0125] below exemplifies the linked services that can be provided after the AI ​​model (835) recognizes panic disorder through inference.

[0126] Contents Voice input When the user feels the initial symptoms of a panic attack, he or she inputs words such as "I feel anxious" or "My heart is beating fast" through his or her smartphone. Data collection and emotional state analysis Wearable devices worn by the user (e.g., smartwatch, smart ring, smart headset, etc.) collect and analyze heart rate, acceleration, and skin conductance data in real time. Panic attack detection The central processing unit analyzes physiological data such as increased heart rate, rapid breathing, and trembling to detect an imminent panic attack. User notification When a panic attack is detected, the system displays a notification on the smartphone screen and vibrates to warn the user that a panic attack is imminent. Response action initiated. The system immediately guides the user through calming techniques, providing visual and easy-to-follow instructions. Breathing exercise guidance Displays instructions to breathe slowly and deeply on the smartphone screen, and also provides breathing exercise guidance through voice. Meditation music playback Plays meditation music or calming background sounds to stabilize the user's heart rate. Calming video playback Plays videos with a calming effect (e.g., natural scenery, sound of waves, etc.) to relieve the user's anxiety. Emergency contact notification Emergency contact set by the user Automatically send text messages or phone calls to notify contacts (family, friends, etc.) of a panic attack. Send location information. Include your current location in emergency contact notifications so you can receive emergency help when needed. Provide additional support. Offer immediate connection to psychological counseling services or mental health professionals, depending on your condition. Monitor your condition. Continuously monitor physiological data until the attack subsides to check your condition. Request user feedback. After an attack ends, request voice or text feedback asking the user about their condition and record this. Record history. Record the user's panic attack history, response methods, effects, etc. and store them in a database. Analyze data. Analyze stored data to identify patterns and triggers of panic attacks and suggest customized preventive measures. Provide regular reports. Provide weekly or monthly reports to the user.Provides personalized advice to help you check the frequency and progression of panic attacks. Provides personalized advice on improving lifestyle habits and stress management based on analysis results. Connects to additional features. Provides integrated support to improve overall mental health by connecting to fitness apps, meditation apps, etc. as needed. Continuous improvement. Based on user feedback and data, the system is continuously improved to provide more effective panic attack response and emotional support.

[0127] FIG. 8d is a drawing for exemplarily explaining a user interface that diagnoses and provides feedback on a user's emotional state in an on-device environment according to one embodiment.

[0128] Referring to FIG. 8d, an AI model (840) included in an electronic device (e.g., electronic device (100) of FIG. 1) that provides AI services in an on-device environment can collect voice data according to a voice conversation during a user's social interaction (e.g., a conversation with a friend). The AI ​​model (820) can analyze the collected voice data to obtain voice pattern data such as intonation, word usage, tremors, or speech speed. The AI ​​model (820) can infer the user's emotional state based on the voice pattern data. For example, the AI ​​model (820) can recommend an appropriate response (e.g., a method of expressing agreement, a language of empathy) to the user in natural language during a conversation based on the user's emotional state. This can enable smooth interaction with others by taking the user's emotional state into consideration. This can provide a reporting function that allows the user to easily check the user's social interaction history.

[0129] The system of the present invention can be applied to AI models that assess job stress. For example, it can collect workplace conversations and biometric signals from smartwatches, smart rings, smart headsets, and other devices. Based on this data, job stress indicators can be scored. Emotional states can be analyzed. If stress levels are high, stress relief techniques (e.g., short breaks, deep breathing exercises) can be suggested, along with advice on improving work efficiency.

[0130] FIG. 9 is a block diagram of an electronic device (901) (e.g., electronic device (440) of FIG. 4) within a network environment (900) according to various embodiments.

[0131] Referring to FIG. 9, in a network environment (900), an electronic device (901) may communicate with an electronic device (902) via a first network (998) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (904) or a server (908) via a second network (999) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (901) may communicate with the electronic device (904) via the server (908). According to one embodiment, the electronic device (901) may include a processor (920), a memory (930), an input module (950), an audio output module (955), a display module (960), an audio module (970), a sensor module (976), an interface (977), a connection terminal (978), a haptic module (979), a camera module (980), a power management module (988), a battery (989), a communication module (990), a subscriber identification module (996), or an antenna module (997). In some embodiments, the electronic device (901) may omit at least one of these components (e.g., the connection terminal (978)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (976), the camera module (980), or the antenna module (997)) may be integrated into one component (e.g., the display module (960)).

[0132] The processor (920) may control at least one other component (e.g., a hardware or software component) of the electronic device (901) connected to the processor (920) by executing, for example, software (e.g., a program (940)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (920) may store a command or data received from another component (e.g., a sensor module (976) or a communication module (990)) in a volatile memory (932), process the command or data stored in the volatile memory (932), and store the resulting data in a non-volatile memory (934). According to one embodiment, the processor (920) may include a main processor (921) (e.g., a central processing unit or an application processor) or a secondary processor (923) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (901) includes a main processor (921) and a secondary processor (923), the secondary processor (923) may be configured to use less power than the main processor (921) or to be specialized for a given function. The secondary processor (923) may be implemented separately from the main processor (921) or as a part thereof.

[0133] The auxiliary processor (923) may control at least a portion of functions or states associated with at least one component (e.g., a display module (960), a sensor module (976), or a communication module (990)) of the electronic device (901), for example, on behalf of the main processor (921) while the main processor (921) is in an inactive (e.g., sleep) state, or together with the main processor (921) while the main processor (921) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (923) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (980) or a communication module (990)). In one embodiment, the auxiliary processor (923) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (901) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (908)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0134] The memory (930) can store various data used by at least one component (e.g., the processor (920) or the sensor module (976)) of the electronic device (901). The data can include, for example, software (e.g., the program (940)) and input data or output data for commands related thereto. The memory (930) can include volatile memory (932) or non-volatile memory (934).

[0135] The program (940) may be stored as software in memory (930) and may include, for example, an operating system (942), middleware (944), or an application (946).

[0136] The input module (950) can receive commands or data to be used in a component of the electronic device (901) (e.g., a processor (920)) from an external source (e.g., a user) of the electronic device (901). The input module (950) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0137] The audio output module (955) can output audio signals to the outside of the electronic device (901). The audio output module (955) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0138] The display module (960) can visually provide information to an external party (e.g., a user) of the electronic device (901). The display module (960) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (960) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0139] The audio module (970) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (970) can acquire sound through the input module (950), output sound through the sound output module (955), or an external electronic device (e.g., electronic device (902)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (901).

[0140] The sensor module (976) can detect the operating status (e.g., power or temperature) of the electronic device (901) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (976) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0141] The interface (977) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (901) with an external electronic device (e.g., the electronic device (902)). In one embodiment, the interface (977) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0142] The connection terminal (978) may include a connector through which the electronic device (901) may be physically connected to an external electronic device (e.g., the electronic device (902)). According to one embodiment, the connection terminal (978) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0143] The haptic module (979) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (979) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0144] The camera module (980) can capture still images and videos. According to one embodiment, the camera module (980) may include one or more lenses, image sensors, image signal processors, or flashes.

[0145] The power management module (988) can manage power supplied to the electronic device (901). According to one embodiment, the power management module (988) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0146] A battery (989) may power at least one component of the electronic device (901). In one embodiment, the battery (989) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0147] The communication module (990) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (901) and an external electronic device (e.g., electronic device (902), electronic device (904), or server (908)), and the performance of communication through the established communication channel. The communication module (990) may operate independently from the processor (920) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (990) may include a wireless communication module (992) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (994) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (904) via a first network (998) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (999) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (992) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (996) to verify or authenticate the electronic device (901) within a communication network such as the first network (998) or the second network (999).

[0148] The wireless communication module (992) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (992) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (992) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (992) can support various requirements specified in the electronic device (901), an external electronic device (e.g., the electronic device (904)), or a network system (e.g., a second network (999)). According to one embodiment, the wireless communication module (992) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

[0149] The antenna module (997) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (997) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (997) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (998) or the second network (999), may be selected from the plurality of antennas by, for example, the communication module (990). A signal or power may be transmitted or received between the communication module (990) and the external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (997).

[0150] According to various embodiments, the antenna module (997) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0151] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0152] According to one embodiment, commands or data may be transmitted or received between the electronic device (901) and an external electronic device (904) via a server (908) connected to a second network (999). Each of the external electronic devices (902, or 1304) may be the same or a different type of device as the electronic device (901). According to one embodiment, all or part of the operations executed in the electronic device (901) may be executed in one or more of the external electronic devices (902, 1304, or 1308). For example, when the electronic device (901) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (901) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (901). The electronic device (901) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (901) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (904) may include an Internet of Things (IoT) device. The server (908) may be an intelligent server using machine learning and / or a neural network. According to one embodiment, the external electronic device (904) or the server (908) may be included in the second network (999).The electronic device (901) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0153] Although not included in the scenario described above, if a user wearing AR glasses receives a real-time surrounding image generation service based on a cloud environment and then enters a location with poor network communication conditions, such as underground, the model transition method proposed in this document may be utilized to provide uninterrupted service.

[0154] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0155] According to one example, the electronic device (100) may include a memory (120) including one or more storage media for storing instructions. The electronic device (100) may include at least one processor (110) including a processing circuit. When the instructions are individually or collectively executed by the at least one processor (110), they may cause the electronic device (100) to perform at least one operation. The at least one operation may include an operation of acquiring biometric data by at least one sensor (170). The at least one operation may include an operation of acquiring language pattern data based on natural language processing of a call voice and / or an outgoing text. The at least one operation may include an operation of acquiring image analysis data based on an analysis of a target image including at least a captured image. The at least one operation may include an operation of generating keywords related to emotions or sentiments based on the acquired biometric data, the acquired language pattern data, or the image analysis data. The at least one action may include an action of analyzing the keywords generated by at least one on-device artificial intelligence (AI) model to infer an emotional or sentimental state. The at least one action may include an action of providing a linked service based on the results of the inference.

[0156] For example, when the instructions are individually or collectively executed by at least one processor (110), they may cause the electronic device (100) to perform an operation of accumulating and storing the acquired biometric data, the acquired language pattern data, or the image analysis data for a specific period of time.

[0157] For example, when the instructions are individually or collectively executed by at least one processor (110), they may cause the electronic device (100) to perform an operation of determining whether to perform an inference about the emotional state or the sentimental state based on the distribution of the generated keywords.

[0158] For example, when the instructions are individually or collectively executed by at least one processor (110), they may cause the electronic device (100) to perform the following operations: acquiring a user profile; and analyzing the acquired user profile to assign weights to the generated keywords.

[0159] In one example, the weights may be used to determine a score for the emotional state or the sentiment state to be obtained by the inference.

[0160] For example, when the instructions are individually or collectively executed by at least one processor (110), the electronic device (100) may be caused to perform an operation of analyzing the generated keywords and determining a score for each emotion or emotional state by the at least one AI model learned through pre-training in the on-device environment.

[0161] For example, when the instructions are individually or collectively executed by at least one processor (110), the electronic device (100) may cause the at least one AI model to perform an operation of inferring an emotional state or affective state based on a score determined for each emotion or sentiment.

[0162] In one example, when the instructions are individually or collectively executed by at least one processor (110), they may cause the electronic device (100) to perform an action of activating a conversation application for interaction with the user and conducting a conversation with the user regarding the results of the inference.

[0163] In one example, the instructions, when individually or collectively executed by at least one processor (110), may cause the electronic device (100) to perform an action that provides a report on the emotion and / or emotional state on a daily, weekly, or monthly basis based on the results of the inference.

[0164] For example, when the instructions are individually or collectively executed by at least one processor (110), they may cause the electronic device (100) to perform an operation that provides a response corresponding to the diagnosis result of the emotion or emotional state based on the result of the inference.

[0165] According to one example, the storage medium (120) may store computer-readable instructions. The instructions, when executed by at least a portion of at least one processor (110) of the electronic device (440), may cause the electronic device (440) to perform at least one operation. The at least one operation may include an operation of acquiring biometric data by at least one sensor (170). The at least one operation may include an operation of acquiring language pattern data based on natural language processing of a voice call and / or a text message. The at least one operation may include an operation of acquiring image analysis data based on an analysis of a target image including at least a photographed image. The at least one operation may include an operation of generating keywords related to an emotion or sentiment based on the acquired biometric data, the acquired language pattern data, or the image analysis data. The at least one operation may include an operation of analyzing the generated keywords in at least one on-device artificial intelligence (AI) model to infer an emotional state or sentimental state. The above at least one action may include an action of providing a linked service based on a result according to the above inference.

[0166] According to an example, the at least one operation may include an operation of accumulating and storing the acquired biometric data, the acquired language pattern data, or the image analysis data for a specific period of time.

[0167] In one example, the at least one action may include an action of determining whether to perform an inference about the emotional state or the sentiment state based on the distribution of the generated keywords.

[0168] In one example, the at least one action may include an action of obtaining a user profile.

[0169] In one example, the at least one action may include an action of analyzing the acquired user profile and assigning weights to the generated keywords.

[0170] In one example, the weights may be used to determine a score for the emotional state or the affective state to be obtained by the inference.

[0171] In one example, the at least one action may include an action of analyzing the generated keywords and determining a score for each emotion or emotional state by the at least one AI model learned through pre-training in an on-device environment.

[0172] In one example, the at least one action may include an action in which the at least one AI model infers an emotional state or emotional state based on a score determined for each emotion or sentiment.

[0173] In one example, the at least one action may include activating a conversation application for interaction with the user to conduct a conversation with the user regarding the results of the inference.

[0174] In one example, the at least one action may include providing a report on emotional and / or affective states on a daily, weekly, or monthly basis based on the results of the inference.

[0175] Depending on the task, the at least one action may include an action that provides a response plan corresponding to the diagnosis result of the emotion or emotional state based on the result of the inference.

[0176] According to an example, the operating method of the electronic device (440) may include an operation of acquiring biometric data by at least one sensor (170). The operating method may include an operation of acquiring language pattern data based on natural language processing of a call voice and / or an outgoing text message. The operating method may include an operation of acquiring image analysis data based on an analysis of a target image including at least a photographed image. The operating method may include an operation of generating keywords related to an emotion or sentiment based on the acquired biometric data, the acquired language pattern data, or the image analysis data. The operating method may include an operation of analyzing the generated keywords in at least one on-device artificial intelligence (AI) model to infer an emotional state or sentimental state. The operating method may include an operation of providing a linked service based on a result according to the inference.

[0177] According to an example, the operating method may include an operation of accumulating and storing the acquired biometric data, the acquired language pattern data, or the image analysis data for a specific period of time.

[0178] In one example, the method may include an operation of determining whether to perform an inference about the emotional state or the sentimental state based on the distribution of the generated keywords.

[0179] In one example, the act of performing the inference may include the act of obtaining a user profile.

[0180] In one example, the act of performing the inference may include an act of analyzing the acquired user profile and assigning weights to the generated keywords.

[0181] In one example, the weights may be used to determine a score for the emotional state or the affective state to be obtained by the inference.

[0182] In one example, the operation of performing the above inference may include an operation of analyzing the generated keywords and determining a score for each emotion or emotional state by at least one AI model learned through pre-training in an on-device environment.

[0183] In one example, the act of performing the inference may include an act of the at least one AI model inferring an emotional state or emotional state based on a score determined for each emotion or sentiment.

[0184] In one example, the action of providing the linked service may include an action of activating a conversation application for interaction with the user to conduct a conversation with the user regarding the result of the inference.

[0185] In one example, the action of providing the above-described linked service may include an action of providing a report on emotional and / or sentimental states on a daily, weekly, or monthly basis based on the results of the above-described inference.

[0186] As an example, the action of providing the above-mentioned linked service may include an action of providing a response plan corresponding to the diagnosis result of an emotion or emotional state based on the result of the above-mentioned inference.

[0187] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0188] It should be understood that the embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to a specific embodiment, but include various modifications, equivalents, or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the item, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0189] The term "module" used in one embodiment of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0190] An embodiment of the present document may be implemented as software including one or more instructions stored in a storage medium (e.g., memory (120)) readable by a machine (e.g., electronic device (440)). For example, a processor (e.g., processor (110)) of the machine (e.g., electronic device (440)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0191] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0192] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (100), A memory (120) including one or more storage media for storing instructions; and At least one processor (110) including a processing circuit, When the above instructions are executed individually or collectively by at least one processor (110), they cause the electronic device (100) to perform at least one operation, At least one of the above actions, An operation of acquiring biometric data by at least one sensor (170); An action to obtain language pattern data based on natural language processing of spoken voice and / or sent text; An operation of obtaining image analysis data based on an analysis of a target image that includes at least a captured image; An operation of generating keywords related to emotions or feelings based on the acquired biometric data, the acquired language pattern data, or the image analysis data; An action of analyzing the keywords generated by at least one on-device artificial intelligence (AI) model to make an inference about an emotional state or sentimental state; and Action to provide linked services based on the results of the above inference An electronic device (100) comprising:

2. In paragraph 1, When the above instructions are individually or collectively executed by at least one processor (110), the electronic device (100) causes: An operation of accumulating and storing the acquired biometric data, the acquired language pattern data, or the image analysis data for a specific period of time. An electronic device (100) that causes the device to perform a function.

3. In paragraph 1 or 2, When the above instructions are individually or collectively executed by at least one processor (110), the electronic device (100) causes: An action to determine whether to perform inference on the emotional state or the sentimental state based on the distribution of the keywords generated above. An electronic device (100) that causes the device to perform a function.

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by at least one processor (110), the electronic device (100) causes: Actions to obtain a user profile; and An action to assign weights to the keywords generated above based on the analysis results of the user profile obtained above. causes it to perform, The electronic device (100) wherein the above weights are used to determine a score for the emotional state or the sentiment state to be obtained by the above reasoning.

5. In paragraph 3 or 4, When the above instructions are individually or collectively executed by at least one processor (110), the electronic device (100) causes: An operation of determining a score for each emotion or emotional state by at least one AI model learned through pre-training in an on-device environment based on the analysis results of the above-mentioned generated keywords; and An operation in which at least one AI model infers the emotional state or the emotional state based on a score determined for each emotion or sentiment. An electronic device (100) that causes the device to perform a function.

6. In any one of paragraphs 1 to 5, When the above instructions are individually or collectively executed by at least one processor (110), the electronic device (100) causes: An action of activating a conversation application for interaction with the user and conducting a conversation with the user regarding the results of the inference. An electronic device (100) that causes the device to perform a function.

7. In any one of paragraphs 1 to 5, When the above instructions are individually or collectively executed by at least one processor (110), the electronic device (100) causes: An action to provide a report on the emotional and / or sentimental state on a daily, weekly, or monthly basis based on the results of the above inference; and An action that provides a response plan corresponding to the diagnosis result of the above emotion and / or emotional state based on the result of the above inference. An electronic device (100) that causes the device to perform a function.

8. In the operating method of the electronic device (440), An operation of acquiring biometric data by at least one sensor (170); An action to obtain language pattern data based on natural language processing of spoken voice and / or sent text; An operation of obtaining image analysis data based on an analysis of a target image that includes at least a captured image; An operation of generating keywords related to emotions or feelings based on the acquired biometric data, the acquired language pattern data, or the image analysis data; An action of analyzing the keywords generated by at least one on-device artificial intelligence (AI) model to make an inference about an emotional state or sentimental state; and Action to provide linked services based on the results of the above inference A method comprising:

9. In paragraph 8, A method comprising an operation of accumulating and storing the acquired biometric data, the acquired language pattern data, or the image analysis data for a specific period of time.

10. In paragraph 8 or 9, A method comprising an operation of determining whether to perform an inference on the emotional state or the sentimental state based on the distribution of the generated keywords.

11. In any one of paragraphs 8 to 10, The action of performing the above inference is, Actions to obtain a user profile; and An action to analyze the user profile obtained above and assign weights to the keywords generated above. Includes, A method wherein the above weights are used to determine a score for the emotional state or the sentiment state to be obtained by the above inference.

12. In paragraph 10 or 11, The action of performing the above inference is, An operation of analyzing the above-generated keywords and determining a score for each emotion or emotional state by at least one AI model learned through pre-training in an on-device environment; and An action of inferring an emotional state or emotional state based on a score determined by the at least one AI model for each emotion or sentiment A method comprising:

13. In any one of paragraphs 8 to 12, The action of providing the above linked service is: An action of activating a conversation application for interaction with the user and conducting a conversation with the user regarding the results of the inference. A method comprising:

14. In any one of paragraphs 8 to 12, The action of providing the above linked service is: An action to provide a report on the emotional and / or sentimental state on a daily, weekly, or monthly basis based on the results of the above inference; and An action that provides a response plan corresponding to the diagnosis result of an emotion or emotional state based on the results of the above inference. A method comprising:

15. In a storage medium (120) that stores instructions that can be read by a computer, The above instructions, when executed by at least a part of at least one processor (110) of the electronic device (100), cause the electronic device (100) to perform at least one operation; At least one of the above actions: An operation of acquiring biometric data by at least one sensor (170); An action to obtain language pattern data based on natural language processing of spoken voice and / or sent text; An operation of obtaining image analysis data based on an analysis of a target image that includes at least a captured image; An operation of generating keywords related to emotions or feelings based on the acquired biometric data, the acquired language pattern data, or the image analysis data; An action of analyzing the keywords generated by at least one on-device artificial intelligence (AI) model to make an inference about an emotional state or sentimental state; and Action to provide linked services based on the results of the above inference A storage medium (120) including:

Images